You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Riza Suminto (Jira)" <ji...@apache.org> on 2025/03/14 18:58:00 UTC
[jira] [Resolved] (IMPALA-13860) DHCECK hit in cluster-membership.mgr.cc
[ https://issues.apache.org/jira/browse/IMPALA-13860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Riza Suminto resolved IMPALA-13860.
-----------------------------------
Fix Version/s: Impala 5.0.0
Resolution: Fixed
> DHCECK hit in cluster-membership.mgr.cc
> ---------------------------------------
>
> Key: IMPALA-13860
> URL: https://issues.apache.org/jira/browse/IMPALA-13860
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> A DHCECK is hit in cluster-membership.mgr.cc when enabling impala graceful shutdown in test_coord_only_pool_exec_groups. The following is the log from crashing impalad during shutdown.
> {noformat}
> I0313 03:34:10.643050 3569793 init.cc:260] Shutdown signal received. Current Shutdown Status: shutdown grace period left: 0, deadline left: 1m, cancel deadline left: 48s000ms, queries registered on coordinator: 0, queries executing: 0, fragment instances: 0
> I0313 03:34:10.685133 3569791 cluster-membership-mgr.cc:247] Processing statestore update
> I0313 03:34:10.685143 3569791 cluster-membership-mgr.cc:248] Local backend membership needs update
> I0313 03:34:10.685145 3569791 cluster-membership-mgr.cc:264] Received delta membership update
> I0313 03:34:10.685168 3569791 cluster-membership-mgr.cc:373] Removing backend 054b8dbd87ae1d41:dea01c0c3357a890 from group name: "root.group-set-small-group-000"
> min_size: 1
> (quiescing)
> I0313 03:34:10.685184 3569791 cluster-membership-mgr.cc:65] Removing empty group name: "root.group-set-small-group-000"
> min_size: 1
> I0313 03:34:10.685212 3569791 cluster-membership-mgr.cc:433] Removing local backend from group name: "root.group-set-small-group-000"
> min_size: 2
> F0313 03:34:10.685215 3569791 cluster-membership-mgr.cc:61] Check failed: it != executor_groups->end()
> Minidump in thread [3569791]StatestoreSubscriber-2 running query 0000000000000000:0000000000000000, fragment instance 0000000000000000:0000000000000000
> Wrote minidump to /data/jenkins/workspace/impala-private-basic-parameterized/repos/Impala/logs/custom_cluster_tests/minidumps/impalad/354840e1-0cb4-41b9-0fb152aa-d09ceb81.dmp{noformat}
> There are two back-to-back calls to
> RemoveExecutorAndGroup().
> The [first call|https://github.com/apache/impala/blob/8093c3fa6b44f7f6ec699d2dd47581401f75f363/be/src/scheduling/cluster-membership-mgr.cc#L375], remove backend 054b8dbd87ae1d41:dea01c0c3357a890 and group "root.group-set-small-group-000" that turns empty after removal of backend 054b8dbd87ae1d41:dea01c0c3357a890 from new_executor_groups.
> The [second call|https://github.com/apache/impala/blob/8093c3fa6b44f7f6ec699d2dd47581401f75f363/be/src/scheduling/cluster-membership-mgr.cc#L433], remove local backend from new_executor_groups, which then hit the DCHECK for not finding the group name "root.group-set-small-group-000".
>
> The DCHECK in RemoveExecutorAndGroup should be replaced by an if and VLOG(1).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)