You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2022/10/31 21:55:00 UTC
[jira] [Comment Edited] (IMPALA-11688) Kudu DCHECK hit in tls_handshake.cc during test_kudu_txn_heartbeat
[ https://issues.apache.org/jira/browse/IMPALA-11688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17626862#comment-17626862 ]
Wenzhe Zhou edited comment on IMPALA-11688 at 10/31/22 9:54 PM:
----------------------------------------------------------------
From stack trace, DCHECK failure happened in Kudu C++ client:
{code:java}
Crash reason: SIGABRT
Crash address: 0x7d100004b7c
Process uptime: not available
Thread 518 (crashed)
0 libc.so.6 + 0x36387
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0xffffffffffffffff rbx = 0x00000000055739a0
rsi = 0x00000000000055e6 rdi = 0x0000000000004b7c
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0608
r8 = 0x0000000000000000 r9 = 0x00007f01246e04c0
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000055739a0 r13 = 0x0000000005573a00
r14 = 0x00000000000000b6 r15 = 0x0000000000000078
rip = 0x00007f02563be387
Found by: given as instruction pointer in context
1 libc.so.6 + 0x37a78
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0610
rip = 0x00007f02563bfa78
Found by: stack scanning
2 impalad!google_breakpad::ExceptionHandler::HandleSignal(int, siginfo_t*, void*) + 0x1a0
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0698
rip = 0x0000000003777210
Found by: stack scanning
3 impalad!google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) [clone .part.0] + 0xac0
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0738
rip = 0x0000000003748600
Found by: stack scanning
4 impalad!google::DumpStackTraceAndExit() [clone .cold] + 0x5
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0740
rip = 0x0000000000e10eb7
Found by: stack scanning
5 impalad!google::LogDestination::WaitForSinks(google::LogMessage::LogMessageData*) + 0x110
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e07c0
rip = 0x0000000003744070
Found by: stack scanning
6 impalad!google::LogMessage::Fail() + 0xd
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e07f0
rip = 0x000000000373d2cd
Found by: stack scanning
7 impalad!google::LogMessage::SendToLog() + 0x244
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0800
rip = 0x000000000373f204
Found by: stack scanning
8 libstdc++.so.6 + 0x12cae4
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0820
rip = 0x00007f0259ad3ae4
Found by: stack scanning
9 impalad!_fini + 0x3e87
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0828
rip = 0x0000000003a3f103
Found by: stack scanning
10 libkudu_client.so.0 + 0x72f9f3
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0838
rip = 0x00007f025819e9f3
Found by: stack scanning
11 impalad!google::LogMessage::Flush() + 0x1ec
rsp = 0x00007f01246e08d0 rip = 0x000000000373ccac
Found by: stack scanning
12 libstdc++.so.6 + 0x12cae4
rsp = 0x00007f01246e0910 rip = 0x00007f0259ad3ae4
Found by: stack scanning
13 libkudu_client.so.0 + 0x72f9f3
rsp = 0x00007f01246e0960 rip = 0x00007f025819e9f3
Found by: stack scanning
14 impalad!google::LogMessage::~LogMessage() + 0x9
rsp = 0x00007f01246e0970 rip = 0x000000000373d029
Found by: stack scanning
15 impalad!impala::LogKuduMessage(void*, kudu::client::KuduLogSeverity, char const*, int, tm const*, char const*, unsigned long) [kudu-util.cc : 125 + 0x9]
rsp = 0x00007f01246e0980 rip = 0x0000000000f7f037
Found by: stack scanning
16 libstdc++.so.6 + 0x11f8e2
rsp = 0x00007f01246e09a0 rip = 0x00007f0259ac68e2
Found by: stack scanning
17 impalad!kudu::client::KuduLoggingFunctionCallback<void*>::Run(kudu::client::KuduLogSeverity, char const*, int, tm const*, char const*, unsigned long) [callbacks.h : 151 + 0xa]
rsp = 0x00007f01246e09e0 rip = 0x0000000000f8024f
Found by: stack scanning
18 libkudu_client.so.0 + 0x2e8819
rsp = 0x00007f01246e0a00 rip = 0x00007f0257d57819
Found by: stack scanning
19 libkudu_client.so.0 + 0x72f9c0
rsp = 0x00007f01246e0a20 rip = 0x00007f025819e9c0
Found by: stack scanning
20 libkudu_client.so.0 + 0x4244e7
rsp = 0x00007f01246e0a60 rip = 0x00007f0257e934e7
Found by: stack scanning
21 libkudu_client.so.0 + 0x72f9c0
rsp = 0x00007f01246e0ab0 rip = 0x00007f025819e9c0
Found by: stack scanning
22 libstdc++.so.6 + 0x13aa48
rsp = 0x00007f01246e0b00 rip = 0x00007f0259ae1a48
Found by: stack scanning
{code}
tls_handshake is used when SSL is enabled. But for the customer cluster test TestKuduTxnKeepalive::test_kudu_txn_heartbeat, the SSL is not enabled for the Impala cluster. It's not clear why tls_handshake was invoked.
The DCHECK failure caused impalad_node1 crash and unreachable, hence test failed.
{code:java}
Failed due to unreachable impalad(s): impala-ec2-centos79-m6i-4xlarge-ondemand-17b7.vpc.cloudera.com:27001
{code}
was (Author: wzhou):
From stack trace, DCHECK failure happened in Kudu C++ client:
{code:java}
Crash reason: SIGABRT
Crash address: 0x7d100004b7c
Process uptime: not available
Thread 518 (crashed)
0 libc.so.6 + 0x36387
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0xffffffffffffffff rbx = 0x00000000055739a0
rsi = 0x00000000000055e6 rdi = 0x0000000000004b7c
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0608
r8 = 0x0000000000000000 r9 = 0x00007f01246e04c0
r10 = 0x0000000000000008 r11 = 0x0000000000000202
r12 = 0x00000000055739a0 r13 = 0x0000000005573a00
r14 = 0x00000000000000b6 r15 = 0x0000000000000078
rip = 0x00007f02563be387
Found by: given as instruction pointer in context
1 libc.so.6 + 0x37a78
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0610
rip = 0x00007f02563bfa78
Found by: stack scanning
2 impalad!google_breakpad::ExceptionHandler::HandleSignal(int, siginfo_t*, void*) + 0x1a0
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0698
rip = 0x0000000003777210
Found by: stack scanning
3 impalad!google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) [clone .part.0] + 0xac0
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0738
rip = 0x0000000003748600
Found by: stack scanning
4 impalad!google::DumpStackTraceAndExit() [clone .cold] + 0x5
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0740
rip = 0x0000000000e10eb7
Found by: stack scanning
5 impalad!google::LogDestination::WaitForSinks(google::LogMessage::LogMessageData*) + 0x110
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e07c0
rip = 0x0000000003744070
Found by: stack scanning
6 impalad!google::LogMessage::Fail() + 0xd
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e07f0
rip = 0x000000000373d2cd
Found by: stack scanning
7 impalad!google::LogMessage::SendToLog() + 0x244
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0800
rip = 0x000000000373f204
Found by: stack scanning
8 libstdc++.so.6 + 0x12cae4
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0820
rip = 0x00007f0259ad3ae4
Found by: stack scanning
9 impalad!_fini + 0x3e87
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0828
rip = 0x0000000003a3f103
Found by: stack scanning
10 libkudu_client.so.0 + 0x72f9f3
rbp = 0x00007f01246e08c0 rsp = 0x00007f01246e0838
rip = 0x00007f025819e9f3
Found by: stack scanning
11 impalad!google::LogMessage::Flush() + 0x1ec
rsp = 0x00007f01246e08d0 rip = 0x000000000373ccac
Found by: stack scanning
12 libstdc++.so.6 + 0x12cae4
rsp = 0x00007f01246e0910 rip = 0x00007f0259ad3ae4
Found by: stack scanning
13 libkudu_client.so.0 + 0x72f9f3
rsp = 0x00007f01246e0960 rip = 0x00007f025819e9f3
Found by: stack scanning
14 impalad!google::LogMessage::~LogMessage() + 0x9
rsp = 0x00007f01246e0970 rip = 0x000000000373d029
Found by: stack scanning
15 impalad!impala::LogKuduMessage(void*, kudu::client::KuduLogSeverity, char const*, int, tm const*, char const*, unsigned long) [kudu-util.cc : 125 + 0x9]
rsp = 0x00007f01246e0980 rip = 0x0000000000f7f037
Found by: stack scanning
16 libstdc++.so.6 + 0x11f8e2
rsp = 0x00007f01246e09a0 rip = 0x00007f0259ac68e2
Found by: stack scanning
17 impalad!kudu::client::KuduLoggingFunctionCallback<void*>::Run(kudu::client::KuduLogSeverity, char const*, int, tm const*, char const*, unsigned long) [callbacks.h : 151 + 0xa]
rsp = 0x00007f01246e09e0 rip = 0x0000000000f8024f
Found by: stack scanning
18 libkudu_client.so.0 + 0x2e8819
rsp = 0x00007f01246e0a00 rip = 0x00007f0257d57819
Found by: stack scanning
19 libkudu_client.so.0 + 0x72f9c0
rsp = 0x00007f01246e0a20 rip = 0x00007f025819e9c0
Found by: stack scanning
20 libkudu_client.so.0 + 0x4244e7
rsp = 0x00007f01246e0a60 rip = 0x00007f0257e934e7
Found by: stack scanning
21 libkudu_client.so.0 + 0x72f9c0
rsp = 0x00007f01246e0ab0 rip = 0x00007f025819e9c0
Found by: stack scanning
22 libstdc++.so.6 + 0x13aa48
rsp = 0x00007f01246e0b00 rip = 0x00007f0259ae1a48
Found by: stack scanning
{code}
tls_handshake is used when SSL is enabled. But for the customer cluster test TestKuduTxnKeepalive::test_kudu_txn_heartbeat, the SSL is not enabled for the Impala cluster. It's not clear why tls_handshake was invoked.
> Kudu DCHECK hit in tls_handshake.cc during test_kudu_txn_heartbeat
> ------------------------------------------------------------------
>
> Key: IMPALA-11688
> URL: https://issues.apache.org/jira/browse/IMPALA-11688
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Assignee: Wenzhe Zhou
> Priority: Major
> Labels: broken-build
>
> Saw this in a recent build:
> {noformat}
> Standard Error
> Log file created at: 2022/10/26 08:23:24
> Running on machine: impala-ec2-centos79-m6i-4xlarge-ondemand-17b7.vpc.cloudera.com
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F1026 08:23:24.180671 21990 tls_handshake.cc:182] Check failed: continue_status.ok() || continue_status.IsIncomplete()
> {noformat}
> It happened during custom_cluster/test_kudu.py::TestKuduTxnKeepalive::test_kudu_txn_heartbeat.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org