You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Petr Fedchenkov (Jira)" <ji...@apache.org> on 2026/10/20 12:31:00 UTC
[jira] [Commented] (IMPALA-13399) Thrift RPCs for Statestore heartbeat timed out randomly in UBSAN build
[ https://issues.apache.org/jira/browse/IMPALA-13399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129113#comment-18129113 ]
Petr Fedchenkov commented on IMPALA-13399:
------------------------------------------
I think I found the root cause of these heartbeat timeouts.
The standby statestored sets {{request_statestore_conn_state}} in its heartbeats, and {{StatestoreSubscriber::Heartbeat()}} answers it with {{GetStatestoreConnState()}} of the subscriber's other {{StatestoreStub}}. That function takes the stub's {{lock_}}, and {{StatestoreStub::RecoveryModeChecker()}} holds the same lock for its whole re-registration loop. So once a subscriber loses the active statestored and enters recovery mode, its answers to the standby statestored's heartbeats wait until the re-registration succeeds, and the heartbeat RPCs time out. A longer {{statestore_heartbeat_tcp_timeout_seconds}} does not help, since the wait is as long as the recovery loop. Because the standby statestored then cannot learn that the subscribers lost the active one, it does not take over; after {{statestore_max_missed_heartbeats}} it can unregister the subscribers, catalogds included.
The logs attached here show this pattern: each subscriber's heartbeat from the standby statestored times out 3 s ({{statestore_heartbeat_tcp_timeout_seconds}}) after that subscriber entered recovery mode:
* impalad :27000 "Connection with statestore lost, entering recovery mode" at 02:18:29.22, standby heartbeat "RPC recv timed out" at 02:18:32.23;
* catalogd :26000 entering recovery mode at 02:18:33.98, heartbeat timeout at 02:18:37.10.
We also hit it on Kubernetes when the active statestored's process was killed every few seconds for 150 s and kept winning the role back: the standby statestored's heartbeats to all 5 subscribers timed out, it unregistered all of them ("No active catalogd available"), and when it took over it designated the other catalogd, so the active catalogd switched without a catalogd failure.
The fix is to not take {{lock_}} in {{GetStatestoreConnState()}} and {{IncCountForUpdateStatestoredRoleRPC()}}: the failure detector has its own lock and the counter is atomic. A patch with a deterministic custom cluster test (the standby statestored is kept from taking over for 90 s after the active one is killed; without the fix it unregisters the subscribers) is posted for review: https://gerrit.cloudera.org/#/c/25151/
> Thrift RPCs for Statestore heartbeat timed out randomly in UBSAN build
> -----------------------------------------------------------------------
>
> Key: IMPALA-13399
> URL: https://issues.apache.org/jira/browse/IMPALA-13399
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Wenzhe Zhou
> Assignee: Wenzhe Zhou
> Priority: Major
> Labels: flaky-test
> Fix For: Impala 4.5.0
>
> Attachments: test_statestore_failover_query_resilience.stacktrace.txt, test_statestore_failover_query_resilience.stderr.txt, test_statestore_failover_query_resilience.tar.gz
>
>
> This issue was saw in IMPALA-13388.
> In UBSAN build, Thrift RPCs for Statestore heartbeat were timed out in Statestored HA unit-tests randomly, and heartbeats were not sent to subscribers. See log messages in statestored log file:
> {code:java}
> I0917 00:06:54.902865 3728987 statestore.cc:1414] Unable to send heartbeat message to subscriber catalog-server@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:26000, received error: RPC recv timed out: dest address: impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23020, rpc: N6impala18THeartbeatResponseE
> Subscriber catalog-server@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:26000 timed-out during heartbeat RPC. Timeout is 3s.
> I0917 00:06:54.902873 3728987 failure-detector.cc:91] 4 consecutive heartbeats failed for 'catalog-server@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:26000'. State is OK
> I0917 00:06:55.382777 3728993 client-cache.h:372] RPC recv timed out: dest address: impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23001, rpc: N6impala18THeartbeatResponseE
> I0917 00:06:55.382797 3728993 client-cache.cc:174] Broken Connection, destroy client for impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23001
> I0917 00:06:55.382866 3728993 statestore.cc:1414] Unable to send heartbeat message to subscriber impalad@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:27001, received error: RPC recv timed out: dest address: impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23001, rpc: N6impala18THeartbeatResponseE
> Subscriber impalad@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:27001 timed-out during heartbeat RPC. Timeout is 3s.
> I0917 00:06:55.382876 3728993 failure-detector.cc:91] 4 consecutive heartbeats failed for 'impalad@impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:27001'. State is OK
> I0917 00:06:56.032806 3728989 client-cache.h:372] RPC recv timed out: dest address: impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23000, rpc: N6impala18THeartbeatResponseE
> I0917 00:06:56.032806 3728990 client-cache.h:372] RPC recv timed out: dest address: impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23002, rpc: N6impala18THeartbeatResponseE
> I0917 00:06:56.032831 3728989 client-cache.cc:174] Broken Connection, destroy client for impala-ec2-rhel88-m7g-4xlarge-ondemand-1aed.vpc.cloudera.com:23000
> {code}
> Subscriber (impalad or catalogd) log files showed that subscribers did not receive heartbeats after successful registrations so that subscribers tried to re-register in the statestored. It's one directional Thrift RPC issue from statestored to subscribers. It seems something wrong in Thrift RPC layer.
> Tried to increase value of flag variable "statestore_heartbeat_tcp_timeout_seconds" from 3 seconds to 6 seconds, the issue still happened.
> This issue did not happen in other type of build, like regular build and ASAN build. Statestored HA unit-tests are not ran for TSAN build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org