[jira] [Created] (HBASE-30245) RPC connection pinned to stale IP after cross-pod IP reuse: NSRE storm persists indefinitely because pooled channel is reused without re-resolving DNS

["samad (Jira)" <ji...@apache.org>]

samad created HBASE-30245:
-----------------------------

             Summary: RPC connection pinned to stale IP after cross-pod IP reuse: NSRE storm persists indefinitely because pooled channel is reused without re-resolving DNS
                 Key: HBASE-30245
                 URL: https://issues.apache.org/jira/browse/HBASE-30245
             Project: HBase
          Issue Type: Bug
         Environment: * HBase 2.5.12 client (present in other client versions as well)
 * HBase deployed on Kubernetes.
 * RegionServer and DataNode are co-hosted in the same pod.
 * Multiple HBase clusters run in the same Kubernetes environment.
            Reporter: samad


Summary: Under a narrow but real Kubernetes condition — a RegionServer pod's IP is reassigned to a different live pod (sometimes a RegionServer of an entirely different HBase cluster) — the async client gets stuck issuing requests to that wrong-but-live server and receives a continuous stream of {{NotServingRegionException}} (NSRE) for the affected regions. The condition does not self-heal: only a client process restart fixes it.



We run HBase on Kubernetes, where pod cohost both a RegionServer and a DataNode. Multiple independent HBase clusters share the same Kubernetes environment.

We observed a failure scenario during node maintenance where an HBase async client can become permanently stuck talking to the wrong RegionServer after Kubernetes pod IP reuse.

Consider the following example:
 * *Pod A* hosts *RegionServer A* belonging to {*}HBase Cluster A{*}.
 * *Pod B* hosts *RegionServer B* belonging to {*}HBase Cluster B{*}.
 * Both pods are running on the same Kubernetes node.

During a maintenance activity (node reboot, drain, upgrade, etc.), all pods on the node restart.

A possible sequence is:
 # Pod A goes down.
 # Kubernetes later reassigns Pod A's old IP address to Pod B.
 # The client already has an established TCP connection to Pod A's old IP.
 # Because the connection remains alive through the networking/service-mesh layer, the client does not see a transport failure.
 # Requests intended for RegionServer A are now delivered to RegionServer B, which belongs to a completely different HBase cluster and has never hosted the requested regions.
 # RegionServer B correctly responds with {{{}NotServingRegionException (NSRE){}}}.
 # The client continues reusing the same underlying RPC connection and it gets following continuously 



2026-06-09T06:03:15.226Z, org.apache.hadoop.hbase.NotServingRegionException: 
org.apache.hadoop.hbase.NotServingRegionException: 
app_ns:my-table,rowprefix-0001,1761217611582.65e37b957f2a12c0c710d3866bece520. 
is not online on hbase-B-dn-4.hbase-B.k8s-namespace.svc.cluster.local,16020,1780977222833
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3552)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3530)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1486)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2972)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
 
Note: The table `app_ns:my-table` belongs to HBase cluster-A. The RegionServer 
named in the NSRE (`hbase-B-dn-4`) belongs to a completely different HBase cluster 
(cluster-B). This table has never existed on cluster-B. The client is reaching 
cluster-B's RegionServer only because cluster-B's pod acquired the IP address that 
previously belonged to cluster-A's pod (hbase-A-dn-7). After a Kubernetes node 
maintenance event, both pods restarted on the same node. Cluster-B's pod came up 
first and was assigned cluster-A's old pod IP. The client's existing TCP channel 
was pinned to that IP and never re-resolved DNS, so all RPCs intended for 
cluster-A's RegionServer are landing on cluster-B's RegionServer instead.
 # Requests continue hitting RegionServer B and receive NSREs indefinitely.

In production, we observed this condition persisting for approximately an hour and generating tens of thousands of NSREs. Recovery occurred only after restarting the hbase client process



*Observed Behavior*
 * Continuous NSREs for the same regions.
 * The same RegionServer appears in all NSRE responses.
 * The responding RegionServer belongs to a different HBase cluster than the target table.
 * No transport errors or connection failures are observed.
 * Client restart immediately restores normal operation.



*Expected Behavior*

When the client receives repeated NSREs from a RegionServer that does not match the expected destination, it should eventually drop the existing connection and establish a fresh one, allowing DNS re-resolution and recovery without requiring a client restart.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)