You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Bakai Ádám (Jira)" <ji...@apache.org> on 2024/12/18 13:26:00 UTC
[jira] [Updated] (KUDU-3634) check error in ksck_results.cc in TSAN build
[ https://issues.apache.org/jira/browse/KUDU-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bakai Ádám updated KUDU-3634:
-----------------------------
Description:
{code:java}
F20241218 13:05:04.229065 4953 ksck_results.cc:160] Check failed: servers.size() <= server_count (6 vs. 3)
*** Check failure stack trace: ***
*** Aborted at 1734527104 (unix time) try "date -d @1734527104" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800001359) received by PID 4953 (TID 0x7f5270649880) from PID 4953; stack trace: ***
@ 0x2d8140 __tsan::CallUserSignalHandler() at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2da623 rtl_sigaction() at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
I20241218 13:05:04.427536 14086 raft_consensus.cc:1270] T 5d3037cff4d843b0a7dbc9ff78a339ba P 3e6bbc8984eb4fbfbefd27d5036db51a [term 5 FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20241218 13:05:04.429109 14549 consensus_queue.cc:1035] T 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host: "127.4.214.67" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
@ 0x7f527808f980 (unknown) at ??:0
I20241218 13:05:04.509820 14545 raft_consensus.cc:491] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20241218 13:05:04.510313 14545 raft_consensus.cc:513] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type: VOTER last_known_addr { host: "127.4.214.65" port: 36387 } } peers { permanent_uuid: "1a37ef9543954c8e823eeae2d8592107" member_type: VOTER last_known_addr { host: "127.4.214.66" port: 36001 } } peers { permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host: "127.4.214.67" port: 37251 } }
I20241218 13:05:04.513247 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba" candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1a37ef9543954c8e823eeae2d8592107" is_pre_election: true
I20241218 13:05:04.516223 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba" candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" is_pre_election: true
I20241218 13:05:04.517762 13891 leader_election.cc:304] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfce15bbb5f4f0aa5a9e60872491f3a; no voters: 1a37ef9543954c8e823eeae2d8592107, 3e6bbc8984eb4fbfbefd27d5036db51a
I20241218 13:05:04.518554 14506 raft_consensus.cc:2743] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20241218 13:05:04.518522 14545 leader_election.cc:290] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1a37ef9543954c8e823eeae2d8592107 (127.4.214.66:36001), 3e6bbc8984eb4fbfbefd27d5036db51a (127.4.214.67:37251)
@ 0x7f5271309fb7 gsignal at ??:0
I20241218 13:05:04.633380 13919 raft_consensus.cc:1270] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20241218 13:05:04.634791 14549 consensus_queue.cc:1035] T 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type: VOTER last_known_addr { host: "127.4.214.65" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7f527130b921 abort at ??:0
@ 0x2de136 abort at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f5273d37d64 google::LogMessage::SendToLog() at ??:0
@ 0x7f5273d38910 google::LogMessage::Flush() at ??:0
@ 0x7f5273d3da4b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f527ec9165b kudu::tools::(anonymous namespace)::ServerCsv() at ??:0
@ 0x7f527ec89f4e kudu::tools::PrintTaggedFlagTable() at ??:0
@ 0x7f527ec8415e kudu::tools::KsckResults::PrintTo() at ??:0
@ 0x7f527eb8e0f9 kudu::tools::Ksck::PrintResults() at ??:0
@ 0x7f527eb8e1a1 kudu::tools::Ksck::RunAndPrintResults() at ??:0
@ 0x388cd8 kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody()::$_10::operator()() at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671 @ 0x388c01 _ZNSt3__18__invokeIRZN4kudu5tools46RemoteKsckTest_TestNonMatchingTableFilter_Test8TestBodyEvE4$_10JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899
@ 0x388b61 std::__1::__invoke_void_return_wrapper<>::__call<>() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348
@ 0x388b01 std::__1::__function::__alloc_func<>::operator()() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557
@ 0x386e30 std::__1::__function::__func<>::operator()() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731
@ 0x7f527f277417 std::__1::__function::__value_func<>::operator()() at ??:0
@ 0x7f527f277129 std::__1::function<>::operator()() at ??:0
W20241218 13:05:06.008596 14053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.214.66:36001, user_credentials={real_user=slave}} blocked reactor thread for 63952.8us
@ 0x7f527f1a4920 kudu::AssertEventually() at ??:0
@ 0x36b1a5 kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody() at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671
@ 0x7f527ed34df0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f527ed129f2 testing::Test::Run() at ??:0
@ 0x7f527ed13d09 testing::TestInfo::Run() at ??:0
@ 0x7f527ed14ce5 testing::TestSuite::Run() at ??:0
@ 0x7f527ed297d5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7f527ed35cb0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7f527ed28d8d testing::UnitTest::Run() at ??:0
@ 0x3ca0b8 RUN_ALL_TESTS() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293
@ 0x3c7f4b main at /root/tmp/test123/kudu/src/kudu/util/test_main.cc:109
{code}
> check error in ksck_results.cc in TSAN build
> --------------------------------------------
>
> Key: KUDU-3634
> URL: https://issues.apache.org/jira/browse/KUDU-3634
> Project: Kudu
> Issue Type: Sub-task
> Reporter: Bakai Ádám
> Priority: Major
>
> {code:java}
> F20241218 13:05:04.229065 4953 ksck_results.cc:160] Check failed: servers.size() <= server_count (6 vs. 3)
> *** Check failure stack trace: ***
> *** Aborted at 1734527104 (unix time) try "date -d @1734527104" if you are using GNU date ***
> PC: @ 0x0 (unknown)
> *** SIGABRT (@0x3e800001359) received by PID 4953 (TID 0x7f5270649880) from PID 4953; stack trace: ***
> @ 0x2d8140 __tsan::CallUserSignalHandler() at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
> @ 0x2da623 rtl_sigaction() at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
> I20241218 13:05:04.427536 14086 raft_consensus.cc:1270] T 5d3037cff4d843b0a7dbc9ff78a339ba P 3e6bbc8984eb4fbfbefd27d5036db51a [term 5 FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
> I20241218 13:05:04.429109 14549 consensus_queue.cc:1035] T 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host: "127.4.214.67" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
> @ 0x7f527808f980 (unknown) at ??:0
> I20241218 13:05:04.509820 14545 raft_consensus.cc:491] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
> I20241218 13:05:04.510313 14545 raft_consensus.cc:513] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type: VOTER last_known_addr { host: "127.4.214.65" port: 36387 } } peers { permanent_uuid: "1a37ef9543954c8e823eeae2d8592107" member_type: VOTER last_known_addr { host: "127.4.214.66" port: 36001 } } peers { permanent_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" member_type: VOTER last_known_addr { host: "127.4.214.67" port: 37251 } }
> I20241218 13:05:04.513247 14001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba" candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "1a37ef9543954c8e823eeae2d8592107" is_pre_election: true
> I20241218 13:05:04.516223 14089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3037cff4d843b0a7dbc9ff78a339ba" candidate_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "3e6bbc8984eb4fbfbefd27d5036db51a" is_pre_election: true
> I20241218 13:05:04.517762 13891 leader_election.cc:304] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0bfce15bbb5f4f0aa5a9e60872491f3a; no voters: 1a37ef9543954c8e823eeae2d8592107, 3e6bbc8984eb4fbfbefd27d5036db51a
> I20241218 13:05:04.518554 14506 raft_consensus.cc:2743] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
> I20241218 13:05:04.518522 14545 leader_election.cc:290] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 1a37ef9543954c8e823eeae2d8592107 (127.4.214.66:36001), 3e6bbc8984eb4fbfbefd27d5036db51a (127.4.214.67:37251)
> @ 0x7f5271309fb7 gsignal at ??:0
> I20241218 13:05:04.633380 13919 raft_consensus.cc:1270] T 5d3037cff4d843b0a7dbc9ff78a339ba P 0bfce15bbb5f4f0aa5a9e60872491f3a [term 5 FOLLOWER]: Refusing update from remote peer 1a37ef9543954c8e823eeae2d8592107: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
> I20241218 13:05:04.634791 14549 consensus_queue.cc:1035] T 5d3037cff4d843b0a7dbc9ff78a339ba P 1a37ef9543954c8e823eeae2d8592107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bfce15bbb5f4f0aa5a9e60872491f3a" member_type: VOTER last_known_addr { host: "127.4.214.65" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
> @ 0x7f527130b921 abort at ??:0
> @ 0x2de136 abort at /root/tmp/test123/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
> @ 0x7f5273d37d64 google::LogMessage::SendToLog() at ??:0
> @ 0x7f5273d38910 google::LogMessage::Flush() at ??:0
> @ 0x7f5273d3da4b google::LogMessageFatal::~LogMessageFatal() at ??:0
> @ 0x7f527ec9165b kudu::tools::(anonymous namespace)::ServerCsv() at ??:0
> @ 0x7f527ec89f4e kudu::tools::PrintTaggedFlagTable() at ??:0
> @ 0x7f527ec8415e kudu::tools::KsckResults::PrintTo() at ??:0
> @ 0x7f527eb8e0f9 kudu::tools::Ksck::PrintResults() at ??:0
> @ 0x7f527eb8e1a1 kudu::tools::Ksck::RunAndPrintResults() at ??:0
> @ 0x388cd8 kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody()::$_10::operator()() at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671 @ 0x388c01 _ZNSt3__18__invokeIRZN4kudu5tools46RemoteKsckTest_TestNonMatchingTableFilter_Test8TestBodyEvE4$_10JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/type_traits:3899
> @ 0x388b61 std::__1::__invoke_void_return_wrapper<>::__call<>() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/__functional_base:348
> @ 0x388b01 std::__1::__function::__alloc_func<>::operator()() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1557
> @ 0x386e30 std::__1::__function::__func<>::operator()() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/c++/v1/functional:1731
> @ 0x7f527f277417 std::__1::__function::__value_func<>::operator()() at ??:0
> @ 0x7f527f277129 std::__1::function<>::operator()() at ??:0
> W20241218 13:05:06.008596 14053 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.214.66:36001, user_credentials={real_user=slave}} blocked reactor thread for 63952.8us
> @ 0x7f527f1a4920 kudu::AssertEventually() at ??:0
> @ 0x36b1a5 kudu::tools::RemoteKsckTest_TestNonMatchingTableFilter_Test::TestBody() at /root/tmp/test123/kudu/src/kudu/tools/ksck_remote-test.cc:671
> @ 0x7f527ed34df0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f527ed129f2 testing::Test::Run() at ??:0
> @ 0x7f527ed13d09 testing::TestInfo::Run() at ??:0
> @ 0x7f527ed14ce5 testing::TestSuite::Run() at ??:0
> @ 0x7f527ed297d5 testing::internal::UnitTestImpl::RunAllTests() at ??:0
> @ 0x7f527ed35cb0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7f527ed28d8d testing::UnitTest::Run() at ??:0
> @ 0x3ca0b8 RUN_ALL_TESTS() at /root/tmp/test123/kudu/thirdparty/installed/tsan/include/gtest/gtest.h:2293
> @ 0x3c7f4b main at /root/tmp/test123/kudu/src/kudu/util/test_main.cc:109
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)