You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by David Smiley <ds...@apache.org> on 2023/11/16 04:29:18 UTC

LBSolrClient and "zombie" check at core level vs node level

With lots of cores on a node, and a node having a network issue (or other
node level issue preventing connectivity), I'm seeing that the SolrJ
LBSolrClient "zombie" tracking is not really scalable.  It works on a core
basis, it doesn't know about nodes.  Essentially a client or a healthy node
tries to reach a core on an unhealthy node, and it fails, and is deemed a
"zombie" (in the code).  Imagine distributed search on a massive many-shard
collection where hundreds of replicas are trying to be reached on this node
at the same time, and failing.  In such a circumstance, I've seen a thread
dump showing a thousand threads (in state RUNNABLE, not blocked) all
attempting to compute zombieServers.keySet().toString() (it's a
ConcurrentHashMap) to produce an exception message.  Yikes!  A quick fix
would be to omit that entirely, maybe if the number of zombies is above
10.  I'll file a JIRA issue.  Ideally the node would be deemed not live and
then it wouldn't be reached again until it returned, but I think in this
case the distributed search fan-out was so massive and thus ~instantaneous
that there was no room for a more graceful transition.

Moreover, I wonder if others have found LBSolrClient to be problematic with
so many cores per node?  I'm seeing that tons of zombie checking queries go
out when there's a wide problem.  I think that LBSolrClient ought to know
about the nodes and should try a node level healthceck ping before
executing any core level requests.  Maybe if the healthcheck failed then
succeeded, and if all of a small sample of zombie cores there pass, assume
they will all pass (don't send pings to all).  Just a rough idea.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

Re: LBSolrClient and "zombie" check at core level vs node level

Posted by Chris Hostetter <ho...@fucit.org>.
: By relying only on ZK we’d lose the ability to react quickly to issues that
: might not even make it to a ZK state change.
: 
: I prefer that we keep the current approach but fix the implementation
: (David’s node health check suggestion seems interesting, as would capping
: the number of pings sent to a given node over a certain duration) rather
: than change the approach completely and lose the ability to quickly isolate
: non functional entities.
: 
: By the way, if LBSolrClient removes from its zombie list replicas marked
: down in ZK (it should) and we still see a large number of pings, it means
: these replicas are still ACTIVE in ZK.
: A Solr Client relying only on ZK state might flood the problematic node
: even more.

All fair points.

Which makes me wonder if the real problem is that zombie *tracking* is 
tightly couple with zombie *checking* ?

IIRC LBSolrClient is already smart enough that it is willing to try to 
send a request to (one or more) zombie URLs as a last resort if the 
other URLs don't work -- and if a zombie server successfully responds 
to a request, LBSolrClient removes that URL from "zombie jail" 
(independent of when/if the "zombie ping check" thread has checked thta 
URL)

So maybe we keep all the "zombie tracking" around, but (make it 
configurable to) remove the zombie ping requests?   

Continue to put URLs in in zombie jail for X seconds, but at the end of X 
seconds we just let them out of jail, w/o any "zombie check" queries.  
Trusting that by the end of X seconds either the pod is back up, or zk has 
been updated to mark it down and CloudSolrClient will stop asking us to 
query it.  And even if neither of those things happen, only one (or a few 
concurrent) failed requests will be needed to put it back in jail (and 
LBSolrClient will retry them on other nodes).


Speaking of which: I haven't reviewed the code thoroughly, but is there 
currently a memory leak in LBSolrClient in a situation where a 
node/replica goes away permentantly as CloudSolrClient is making a request 
via LBSolrClient with that URL ? ... doesn't the current code add it to 
the zombie list and then just keep it there forever? (with the background 
thread constantly pinging it?)


And just to clarify: I'm not making these strawman proposals because I 
object to adding node level zombie checks before replica level -- i'm just 
trying to think about the problem in way that might *reduce* the total 
number of requests to solr nodes that (may be) struggling before jumping 
in to solutions that by definition increasing them.





-Hoss
http://www.lucidworks.com/

Re: LBSolrClient and "zombie" check at core level vs node level

Posted by Ilan Ginzburg <il...@gmail.com>.
I believe state in ZK should be used to decide that an entity is
down/unreachable, but given updates to entity states is slow requiring
session expiration first, then the time for the watches to notify, it
cannot IMO be the only parameter.

The proof of the API is in the call (pudding analogy).

I find sound the existing approach in which ZK provides a set of targets to
try, but these destinations are considered temporarily unreachable if they
don’t do the job regardless of what ZK says.

By relying only on ZK we’d lose the ability to react quickly to issues that
might not even make it to a ZK state change.

I prefer that we keep the current approach but fix the implementation
(David’s node health check suggestion seems interesting, as would capping
the number of pings sent to a given node over a certain duration) rather
than change the approach completely and lose the ability to quickly isolate
non functional entities.

By the way, if LBSolrClient removes from its zombie list replicas marked
down in ZK (it should) and we still see a large number of pings, it means
these replicas are still ACTIVE in ZK.
A Solr Client relying only on ZK state might flood the problematic node
even more.

Ilan

On Mon 20 Nov 2023 at 16:07, David Smiley <ds...@apache.org> wrote:

> That's a really fine idea Hoss!
>
> After reviewing LBSolrClient again, I think your proposal would best be a
> new SolrClient subclass.  LBSolrClient has a fair amount of state tracking
> but a failover-only client would track no state.  Perhaps LBSolrClient
> might subclass it or not.
>
> After some discussion with my colleagues, we might try an experiment that
> attempts this and see how it goes.  I suspect it'll be a net positive.  It
> takes some production bake time to really get confidence in something of
> this nature.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Thu, Nov 16, 2023 at 1:55 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
> >
> > I think it's worth rememberinbg that LBSolrClient, and it's design,
> > pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes
> > & replicas are "live" ... it was written at a time when people had to
> > manually specify the list of solr servers and cores themselve when
> sending
> > requests.
> >
> > Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was
> > wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea
> what
> > nodes & replicas are "live" when it sends the request, but LBSolrClient
> > doesn't so...
> >
> > : out when there's a wide problem.  I think that LBSolrClient ought to
> know
> > : about the nodes and should try a node level healthceck ping before
> > : executing any core level requests.  Maybe if the healthcheck failed
> then
> > : succeeded, and if all of a small sample of zombie cores there pass,
> > assume
> > : they will all pass (don't send pings to all).  Just a rough idea.
> >
> > ...i think it's worth considering an inverse idea: make it configurable
> > (and probably change the default given the common usecase is SolrCloud)
> to
> > build a LBSolrClient that does *NO* zombie tracking at all -- it just
> > continues to use the multiple URL options it's given for each request to
> > retry on (certain types of failures).
> >
> > Leave the "live" node/replica tracking to the CloudSolrClient layer, and
> > if there are code paths where it's possible CloudSolrClient is pasing
> > stale lists of replica URLs to LBSolrClient that it (should) already know
> > are not alive (via zk watchers), let's treat those as bugs in
> > CloudSolrClient and fix them.
> >
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>

Re: LBSolrClient and "zombie" check at core level vs node level

Posted by David Smiley <ds...@apache.org>.
That's a really fine idea Hoss!

After reviewing LBSolrClient again, I think your proposal would best be a
new SolrClient subclass.  LBSolrClient has a fair amount of state tracking
but a failover-only client would track no state.  Perhaps LBSolrClient
might subclass it or not.

After some discussion with my colleagues, we might try an experiment that
attempts this and see how it goes.  I suspect it'll be a net positive.  It
takes some production bake time to really get confidence in something of
this nature.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Thu, Nov 16, 2023 at 1:55 PM Chris Hostetter <ho...@fucit.org>
wrote:

>
> I think it's worth rememberinbg that LBSolrClient, and it's design,
> pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes
> & replicas are "live" ... it was written at a time when people had to
> manually specify the list of solr servers and cores themselve when sending
> requests.
>
> Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was
> wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea what
> nodes & replicas are "live" when it sends the request, but LBSolrClient
> doesn't so...
>
> : out when there's a wide problem.  I think that LBSolrClient ought to know
> : about the nodes and should try a node level healthceck ping before
> : executing any core level requests.  Maybe if the healthcheck failed then
> : succeeded, and if all of a small sample of zombie cores there pass,
> assume
> : they will all pass (don't send pings to all).  Just a rough idea.
>
> ...i think it's worth considering an inverse idea: make it configurable
> (and probably change the default given the common usecase is SolrCloud) to
> build a LBSolrClient that does *NO* zombie tracking at all -- it just
> continues to use the multiple URL options it's given for each request to
> retry on (certain types of failures).
>
> Leave the "live" node/replica tracking to the CloudSolrClient layer, and
> if there are code paths where it's possible CloudSolrClient is pasing
> stale lists of replica URLs to LBSolrClient that it (should) already know
> are not alive (via zk watchers), let's treat those as bugs in
> CloudSolrClient and fix them.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>

Re: LBSolrClient and "zombie" check at core level vs node level

Posted by Chris Hostetter <ho...@fucit.org>.
I think it's worth rememberinbg that LBSolrClient, and it's design, 
pre-dates SolrCloud and all of the ZK plumbing we have to know when nodes 
& replicas are "live" ... it was written at a time when people had to 
manually specify the list of solr servers and cores themselve when sending 
requests.

Then when SolrCloud was added, the "zk aware" CloudSolrClient logic was 
wrapped ARROUND LBSolrClient -- CloudSolrClient already has some idea what 
nodes & replicas are "live" when it sends the request, but LBSolrClient 
doesn't so...

: out when there's a wide problem.  I think that LBSolrClient ought to know
: about the nodes and should try a node level healthceck ping before
: executing any core level requests.  Maybe if the healthcheck failed then
: succeeded, and if all of a small sample of zombie cores there pass, assume
: they will all pass (don't send pings to all).  Just a rough idea.

...i think it's worth considering an inverse idea: make it configurable 
(and probably change the default given the common usecase is SolrCloud) to 
build a LBSolrClient that does *NO* zombie tracking at all -- it just 
continues to use the multiple URL options it's given for each request to 
retry on (certain types of failures).

Leave the "live" node/replica tracking to the CloudSolrClient layer, and 
if there are code paths where it's possible CloudSolrClient is pasing 
stale lists of replica URLs to LBSolrClient that it (should) already know 
are not alive (via zk watchers), let's treat those as bugs in 
CloudSolrClient and fix them.



-Hoss
http://www.lucidworks.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org