You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2021/04/09 00:35:28 UTC
[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17293
Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................
KUDU-3268: fix race between MM scheduling and unregistration
Following commit 9e4664d the following race was possible (S: scheduler thread,
T: another thread):
S: finds best op is op1
T: unregisters op1 because the tablet is being shut down
T: takes running_instances_lock_
- waits for running count to go to 0
- destructs op1
S: takes running_instances_lock_
- increments running count to 1
S: attempts to run op1 but segfaults
This resulted in a crash like the following, when shutting down the tserver
(though it seems possible this could manifest itself when deleting a
TabletReplica):
PC: @ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp()
*** SIGSEGV (@0x30) received by PID 21067 (TID 0x7ff96343b700) from PID 48; stack trace: ***
@ 0x7ff976846980 (unknown) at ??:0
@ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7ff97596b538 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7ff97596f124 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff975a05e6e kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff975a06757 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff975a07e7b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff9759f8913 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff97683b6db start_thread at ??:0
@ 0x7ff97388e71f clone at ??:0
This patch fixes this by checking whether the op has been cancelled, while
holding running_instanes_lock_, before proceeding with the scheduling. I added
a new test that failed consistently within 100 runs, that passed 5000 times
locally.
Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Reviewed-on: http://gerrit.cloudera.org:8080/17291
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
(cherry picked from commit 6aeb466c538d8cc566031c4a82200f18f088e18a)
---
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
2 files changed, 25 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/17293/1
--
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17293 )
Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................
KUDU-3268: fix race between MM scheduling and unregistration
Following commit 9e4664d the following race was possible (S: scheduler thread,
T: another thread):
S: finds best op is op1
T: unregisters op1 because the tablet is being shut down
T: takes running_instances_lock_
- waits for running count to go to 0
- destructs op1
S: takes running_instances_lock_
- increments running count to 1
S: attempts to run op1 but segfaults
This resulted in a crash like the following, when shutting down the tserver
(though it seems possible this could manifest itself when deleting a
TabletReplica):
PC: @ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp()
*** SIGSEGV (@0x30) received by PID 21067 (TID 0x7ff96343b700) from PID 48; stack trace: ***
@ 0x7ff976846980 (unknown) at ??:0
@ 0x7ff97596de0d kudu::MaintenanceManager::LaunchOp() at ??:0
@ 0x7ff97596b538 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
@ 0x7ff97596f124 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff975a05e6e kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff975a06757 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff975a07e7b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff977e2bcf4 std::function<>::operator()() at ??:0
@ 0x7ff9759f8913 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff97683b6db start_thread at ??:0
@ 0x7ff97388e71f clone at ??:0
This patch fixes this by checking whether the op has been cancelled, while
holding running_instanes_lock_, before proceeding with the scheduling. I added
a new test that failed consistently within 100 runs, that passed 5000 times
locally.
Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Reviewed-on: http://gerrit.cloudera.org:8080/17291
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
(cherry picked from commit 6aeb466c538d8cc566031c4a82200f18f088e18a)
Reviewed-on: http://gerrit.cloudera.org:8080/17293
---
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
2 files changed, 25 insertions(+), 2 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved
Andrew Wong: Verified
--
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17293 )
Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 09 Apr 2021 00:37:43 +0000
Gerrit-HasComments: No
[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has removed a vote on this change.
Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17293 )
Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................
Patch Set 1: Verified+1
Test failures in RELEASE and DEBUG mode seem to be unrelated (python issues).
--
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 10 Apr 2021 00:03:04 +0000
Gerrit-HasComments: No