You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2023/11/16 04:29:18 UTC
LBSolrClient and "zombie" check at core level vs node level
With lots of cores on a node, and a node having a network issue (or other
node level issue preventing connectivity), I'm seeing that the SolrJ
LBSolrClient "zombie" tracking is not really scalable. It works on a core
basis, it doesn't know about nodes. Essentially a client or a healthy node
tries to reach a core on an unhealthy node, and it fails, and is deemed a
"zombie" (in the code). Imagine distributed search on a massive many-shard
collection where hundreds of replicas are trying to be reached on this node
at the same time, and failing. In such a circumstance, I've seen a thread
dump showing a thousand threads (in state RUNNABLE, not blocked) all
attempting to compute zombieServers.keySet().toString() (it's a
ConcurrentHashMap) to produce an exception message. Yikes! A quick fix
would be to omit that entirely, maybe if the number of zombies is above
10. I'll file a JIRA issue. Ideally the node would be deemed not live and
then it wouldn't be reached again until it returned, but I think in this
case the distributed search fan-out was so massive and thus ~instantaneous
that there was no room for a more graceful transition.
Moreover, I wonder if others have found LBSolrClient to be problematic with
so many cores per node? I'm seeing that tons of zombie checking queries go
out when there's a wide problem. I think that LBSolrClient ought to know
about the nodes and should try a node level healthceck ping before
executing any core level requests. Maybe if the healthcheck failed then
succeeded, and if all of a small sample of zombie cores there pass, assume
they will all pass (don't send pings to all). Just a rough idea.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
Re: LBSolrClient and "zombie" check at core level vs node level
Posted by Chris Hostetter <ho...@fucit.org>.
: By relying only on ZK we’d lose the ability to react quickly to issues that
: might not even make it to a ZK state change.
:
: I prefer that we keep the current approach but fix the implementation
: (David’s node health check suggestion seems interesting, as would capping
: the number of pings sent to a given node over a certain duration) rather
: than change the approach completely and lose the ability to quickly isolate
: non functional entities.
:
: By the way, if LBSolrClient removes from its zombie list replicas marked
: down in ZK (it should) and we still see a large number of pings, it means
: these replicas are still ACTIVE in ZK.
: A Solr Client relying only on ZK state might flood the problematic node
: even more.
All fair points.
Which makes me wonder if the real problem is that zombie *tracking* is
tightly couple with zombie *checking* ?
IIRC LBSolrClient is already smart enough that it is willing to try to
send a request to (one or more) zombie URLs as a last resort if the
other URLs don't work -- and if a zombie server successfully responds
to a request, LBSolrClient removes that URL from "zombie jail"
(independent of when/if the "zombie ping check" thread has checked thta
URL)
So maybe we keep all the "zombie tracking" around, but (make it
configurable to) remove the zombie ping requests?
Continue to put URLs in in zombie jail for X seconds, but at the end of X
seconds we just let them out of jail, w/o any "zombie check" queries.
Trusting that by the end of X seconds either the pod is back up, or zk has
been updated to mark it down and CloudSolrClient will stop asking us to
query it. And even if neither of those things happen, only one (or a few
concurrent) failed requests will be needed to put it back in jail (and
LBSolrClient will retry them on other nodes).
Speaking of which: I haven't reviewed the code thoroughly, but is there
currently a memory leak in LBSolrClient in a situation where a
node/replica goes away permentantly as CloudSolrClient is making a request
via LBSolrClient with that URL ? ... doesn't the current code add it to
the zombie list and then just keep it there forever? (with the background
thread constantly pinging it?)
And just to clarify: I'm not making these strawman proposals because I
object to adding node level zombie checks before replica level -- i'm just
trying to think about the problem in way that might *reduce* the total
number of requests to solr nodes that (may be) struggling before jumping
in to solutions that by definition increasing them.
-Hoss
http://www.lucidworks.com/
Re: LBSolrClient and "zombie" check at core level vs node level
Posted by Ilan Ginzburg <il...@gmail.com>.
I believe state in ZK should be used to decide that an entity is
down/unreachable, but given updates to entity states is slow requiring
session expiration first, then the time for the watches to notify, it
cannot IMO be the only parameter.
The proof of the API is in the call (pudding analogy).
I find sound the existing approach in which ZK provides a set of targets to
try, but these destinations are considered temporarily unreachable if they
don’t do the job regardless of what ZK says.
By relying only on ZK we’d lose the ability to react quickly to issues that
might not even make it to a ZK state change.
I prefer that we keep the current approach but fix the implementation
(David’s node health check suggestion seems interesting, as would capping
the number of pings sent to a given node over a certain duration) rather
than change the approach completely and lose the ability to quickly isolate
non functional entities.
By the way, if LBSolrClient removes from its zombie list replicas marked
down in ZK (it should) and we still see a large number of pings, it means
these replicas are still ACTIVE in ZK.
A Solr Client relying only on ZK state might flood the problematic node
even more.
Ilan
On Mon 20 Nov 2023 at 16:07, David Smiley <ds...@apache.org> wrote:
> That's a really fine idea Hoss!
>
> After reviewing LBSolrClient again, I think your proposal would best be a
> new SolrClient subclass. LBSolrClient has a fair amount of state tracking
> but a failover-only client would track no state. Perhaps LBSolrClient
> might subclass it or not.
>
> After some discussion with my colleagues, we might try an experiment that
> attempts this and see how it goes. I suspect it'll be a net positive. It
> takes some production bake time to really get confidence in something of
> this nature.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Nov 16, 2023 at 1:55 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
> >
> > I think it's worth rememberinbg that LBSolrClient, and it's design,
> > pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes
> > & replicas are "live" ... it was written at a time when people had to
> > manually specify the list of solr servers and cores themselve when
> sending
> > requests.
> >
> > Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was
> > wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea
> what
> > nodes & replicas are "live" when it sends the request, but LBSolrClient
> > doesn't so...
> >
> > : out when there's a wide problem. I think that LBSolrClient ought to
> know
> > : about the nodes and should try a node level healthceck ping before
> > : executing any core level requests. Maybe if the healthcheck failed
> then
> > : succeeded, and if all of a small sample of zombie cores there pass,
> > assume
> > : they will all pass (don't send pings to all). Just a rough idea.
> >
> > ...i think it's worth considering an inverse idea: make it configurable
> > (and probably change the default given the common usecase is SolrCloud)
> to
> > build a LBSolrClient that does *NO* zombie tracking at all -- it just
> > continues to use the multiple URL options it's given for each request to
> > retry on (certain types of failures).
> >
> > Leave the "live" node/replica tracking to the CloudSolrClient layer, and
> > if there are code paths where it's possible CloudSolrClient is pasing
> > stale lists of replica URLs to LBSolrClient that it (should) already know
> > are not alive (via zk watchers), let's treat those as bugs in
> > CloudSolrClient and fix them.
> >
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
Re: LBSolrClient and "zombie" check at core level vs node level
Posted by David Smiley <ds...@apache.org>.
That's a really fine idea Hoss!
After reviewing LBSolrClient again, I think your proposal would best be a
new SolrClient subclass. LBSolrClient has a fair amount of state tracking
but a failover-only client would track no state. Perhaps LBSolrClient
might subclass it or not.
After some discussion with my colleagues, we might try an experiment that
attempts this and see how it goes. I suspect it'll be a net positive. It
takes some production bake time to really get confidence in something of
this nature.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Thu, Nov 16, 2023 at 1:55 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> I think it's worth rememberinbg that LBSolrClient, and it's design,
> pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes
> & replicas are "live" ... it was written at a time when people had to
> manually specify the list of solr servers and cores themselve when sending
> requests.
>
> Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was
> wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea what
> nodes & replicas are "live" when it sends the request, but LBSolrClient
> doesn't so...
>
> : out when there's a wide problem. I think that LBSolrClient ought to know
> : about the nodes and should try a node level healthceck ping before
> : executing any core level requests. Maybe if the healthcheck failed then
> : succeeded, and if all of a small sample of zombie cores there pass,
> assume
> : they will all pass (don't send pings to all). Just a rough idea.
>
> ...i think it's worth considering an inverse idea: make it configurable
> (and probably change the default given the common usecase is SolrCloud) to
> build a LBSolrClient that does *NO* zombie tracking at all -- it just
> continues to use the multiple URL options it's given for each request to
> retry on (certain types of failures).
>
> Leave the "live" node/replica tracking to the CloudSolrClient layer, and
> if there are code paths where it's possible CloudSolrClient is pasing
> stale lists of replica URLs to LBSolrClient that it (should) already know
> are not alive (via zk watchers), let's treat those as bugs in
> CloudSolrClient and fix them.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: LBSolrClient and "zombie" check at core level vs node level
Posted by Chris Hostetter <ho...@fucit.org>.
I think it's worth rememberinbg that LBSolrClient, and it's design,
pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes
& replicas are "live" ... it was written at a time when people had to
manually specify the list of solr servers and cores themselve when sending
requests.
Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was
wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea what
nodes & replicas are "live" when it sends the request, but LBSolrClient
doesn't so...
: out when there's a wide problem. I think that LBSolrClient ought to know
: about the nodes and should try a node level healthceck ping before
: executing any core level requests. Maybe if the healthcheck failed then
: succeeded, and if all of a small sample of zombie cores there pass, assume
: they will all pass (don't send pings to all). Just a rough idea.
...i think it's worth considering an inverse idea: make it configurable
(and probably change the default given the common usecase is SolrCloud) to
build a LBSolrClient that does *NO* zombie tracking at all -- it just
continues to use the multiple URL options it's given for each request to
retry on (certain types of failures).
Leave the "live" node/replica tracking to the CloudSolrClient layer, and
if there are code paths where it's possible CloudSolrClient is pasing
stale lists of replica URLs to LBSolrClient that it (should) already know
are not alive (via zk watchers), let's treat those as bugs in
CloudSolrClient and fix them.
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org