You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Adar Dembo (Jira)" <ji...@apache.org> on 2019/11/01 00:26:00 UTC
[jira] [Resolved] (KUDU-2949) Thread collision in consensus
current_term
[ https://issues.apache.org/jira/browse/KUDU-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adar Dembo resolved KUDU-2949.
------------------------------
Fix Version/s: 1.12.0
Resolution: Fixed
Fixed in commit 87d90b2.
> Thread collision in consensus current_term
> ------------------------------------------
>
> Key: KUDU-2949
> URL: https://issues.apache.org/jira/browse/KUDU-2949
> Project: Kudu
> Issue Type: Bug
> Reporter: Andrew Wong
> Assignee: Adar Dembo
> Priority: Major
> Fix For: 1.12.0
>
> Attachments: raft_consensus_quorum-test.01.txt.xz, raft_consensus_quorum-test.txt
>
>
> Saw a precommit failure of raft_consensus_quorum-test with the following stack trace:
> {code}
> roperty violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
> I0920 01:06:49.326063 626 consensus_queue.cc:984] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
> I0920 01:06:49.326789 624 consensus_queue.cc:984] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
> I0920 01:06:49.357350 233 raft_consensus.cc:2311] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 938a40f5da7f4e62b2c317ecb1ba184c for term 2 because replica is either leader or believes a valid leader to be alive.
> I0920 01:06:49.357631 233 raft_consensus.cc:2890] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 1 FOLLOWER]: Advancing to term 2
> W0920 01:06:50.192647 233 consensus_meta.cc:220] T TestTablet P fa4335770e644e66a6ce962fb388c6f6: Time spent flushing consensus metadata: real 0.835s user 0.004s sys 0.000s
> I0920 01:06:50.192862 233 raft_consensus.cc:2350] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 938a40f5da7f4e62b2c317ecb1ba184c in term 2.
> F0920 01:06:50.193255 233 thread_collision_warner.cc:23] Thread Collision! Previous thread id: 635, current thread id: 233
> *** Check failure stack trace: ***
> I0920 01:06:50.193334 635 raft_consensus.cc:1149] T TestTablet P fa4335770e644e66a6ce962fb388c6f6 [term 2 FOLLOWER]: Rejecting Update request from peer 6fec94ba5fb343fbad900ca4ef38b251 for earlier term 1. Current term is 2. Ops: []
> *** Aborted at 1568941610 (unix time) try "date -d @1568941610" if you are using GNU date ***
> I0920 01:06:50.195124 634 consensus_queue.cc:995] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 }, Status: INVALID_TERM, Last received: 1.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
> I0920 01:06:50.197060 634 raft_consensus.cc:2885] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 LEADER]: Stepping down as leader of term 1
> I0920 01:06:50.197245 634 raft_consensus.cc:705] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6fec94ba5fb343fbad900ca4ef38b251, State: Running, Role: LEADER
> I0920 01:06:50.197593 634 consensus_queue.cc:229] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 peers { permanent_uuid: "938a40f5da7f4e62b2c317ecb1ba184c" member_type: VOTER last_known_addr { host: "peer-0.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "fa4335770e644e66a6ce962fb388c6f6" member_type: VOTER last_known_addr { host: "peer-1.fake-domain-for-tests" port: 0 } } peers { permanent_uuid: "6fec94ba5fb343fbad900ca4ef38b251" member_type: VOTER last_known_addr { host: "peer-2.fake-domain-for-tests" port: 0 } }
> PC: @ 0x7ff647119c37 gsignal
> I0920 01:06:50.198277 634 raft_consensus.cc:2890] T TestTablet P 6fec94ba5fb343fbad900ca4ef38b251 [term 1 FOLLOWER]: Advancing to term 2
> *** SIGABRT (@0x3e8000000e9) received by PID 233 (TID 0x7ff653518bc0) from PID 233; stack trace: ***
> @ 0x45b7bd __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1908
> @ 0x45c7ab rtl_sigaction() at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1997
> @ 0x7ff647f2b330 (unknown) at ??:0
> @ 0x7ff647119c37 gsignal at ??:0
> @ 0x7ff64711d028 abort at ??:0
> @ 0x45e890 __interceptor_abort at /home/jenkins-slave/workspace/kudu-master/1/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1763
> @ 0x7ff649bfdac2 google::logging_fail() at ??:0
> @ 0x7ff649bfc905 google::LogMessage::SendToLog() at ??:0
> @ 0x7ff649bfd3d2 google::LogMessage::Flush() at ??:0
> @ 0x7ff649c01c0b google::LogMessageFatal::~LogMessageFatal() at ??:0
> @ 0x7ff649e8ed25 base::DCheckAsserter::warn() at ??:0
> @ 0x7ff649e8ed9e base::ThreadCollisionWarner::EnterSelf() at ??:0
> @ 0x7ff6524ad7b9 base::ThreadCollisionWarner::ScopedRecursiveCheck::ScopedRecursiveCheck() at ??:0
> @ 0x7ff6524ab582 kudu::consensus::ConsensusMetadata::current_term() at ??:0
> @ 0x4ffcc1 kudu::consensus::RaftConsensusQuorumTest::AssertDurableTermAndVote() at /home/jenkins-slave/workspace/kudu-master/1/src/kudu/consensus/raft_consensus_quorum-test.cc:554
> @ 0x4f6e9b kudu::consensus::RaftConsensusQuorumTest_TestRequestVote_Test::TestBody() at /home/jenkins-slave/workspace/kudu-master/1/src/kudu/consensus/raft_consensus_quorum-test.cc:?
> @ 0x7ff6527c72f0 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7ff6527a64b9 testing::Test::Run() at ??:0
> @ 0x7ff6527a774d testing::TestInfo::Run() at ??:0
> @ 0x7ff6527a8227 testing::TestCase::Run() at ??:0
> @ 0x7ff6527b45fb testing::internal::UnitTestImpl::RunAllTests() at ??:0
> @ 0x7ff6527c8260 testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
> @ 0x7ff6527b3ee3 testing::UnitTest::Run() at ??:0
> @ 0x7ff65311c51c RUN_ALL_TESTS() at ??:0
> @ 0x7ff65311bcc7 main at ??:0
> @ 0x7ff647104f45 __libc_start_main at ??:0
> @ 0x4540f6 (unknown) at ??:?
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)