You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2021/04/09 00:35:28 UTC

[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration

Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17293


Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................

KUDU-3268: fix race between MM scheduling and unregistration

Following commit 9e4664d the following race was possible (S: scheduler thread,
T: another thread):
  S: finds best op is op1
  T: unregisters op1 because the tablet is being shut down
  T: takes running_instances_lock_
    - waits for running count to go to 0
    - destructs op1
  S: takes running_instances_lock_
    - increments running count to 1
  S: attempts to run op1 but segfaults

This resulted in a crash like the following, when shutting down the tserver
(though it seems possible this could manifest itself when deleting a
TabletReplica):
PC: @     0x7ff97596de0d kudu::MaintenanceManager::LaunchOp()
*** SIGSEGV (@0x30) received by PID 21067 (TID 0x7ff96343b700) from PID 48; stack trace: ***
    @     0x7ff976846980 (unknown) at ??:0
    @     0x7ff97596de0d kudu::MaintenanceManager::LaunchOp() at ??:0
    @     0x7ff97596b538 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
    @     0x7ff97596f124 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x7ff977e2bcf4 std::function<>::operator()() at ??:0
    @     0x7ff975a05e6e kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7ff975a06757 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
    @     0x7ff975a07e7b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x7ff977e2bcf4 std::function<>::operator()() at ??:0
    @     0x7ff9759f8913 kudu::Thread::SuperviseThread() at ??:0
    @     0x7ff97683b6db start_thread at ??:0
    @     0x7ff97388e71f clone at ??:0

This patch fixes this by checking whether the op has been cancelled, while
holding running_instanes_lock_, before proceeding with the scheduling. I added
a new test that failed consistently within 100 runs, that passed 5000 times
locally.

Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Reviewed-on: http://gerrit.cloudera.org:8080/17291
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
(cherry picked from commit 6aeb466c538d8cc566031c4a82200f18f088e18a)
---
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
2 files changed, 25 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/17293/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>

[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17293 )

Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................

KUDU-3268: fix race between MM scheduling and unregistration

Following commit 9e4664d the following race was possible (S: scheduler thread,
T: another thread):
  S: finds best op is op1
  T: unregisters op1 because the tablet is being shut down
  T: takes running_instances_lock_
    - waits for running count to go to 0
    - destructs op1
  S: takes running_instances_lock_
    - increments running count to 1
  S: attempts to run op1 but segfaults

This resulted in a crash like the following, when shutting down the tserver
(though it seems possible this could manifest itself when deleting a
TabletReplica):
PC: @     0x7ff97596de0d kudu::MaintenanceManager::LaunchOp()
*** SIGSEGV (@0x30) received by PID 21067 (TID 0x7ff96343b700) from PID 48; stack trace: ***
    @     0x7ff976846980 (unknown) at ??:0
    @     0x7ff97596de0d kudu::MaintenanceManager::LaunchOp() at ??:0
    @     0x7ff97596b538 _ZZN4kudu18MaintenanceManager18RunSchedulerThreadEvENKUlvE_clEv at ??:0
    @     0x7ff97596f124 _ZNSt17_Function_handlerIFvvEZN4kudu18MaintenanceManager18RunSchedulerThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x7ff977e2bcf4 std::function<>::operator()() at ??:0
    @     0x7ff975a05e6e kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7ff975a06757 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
    @     0x7ff975a07e7b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
    @     0x7ff977e2bcf4 std::function<>::operator()() at ??:0
    @     0x7ff9759f8913 kudu::Thread::SuperviseThread() at ??:0
    @     0x7ff97683b6db start_thread at ??:0
    @     0x7ff97388e71f clone at ??:0

This patch fixes this by checking whether the op has been cancelled, while
holding running_instanes_lock_, before proceeding with the scheduling. I added
a new test that failed consistently within 100 runs, that passed 5000 times
locally.

Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Reviewed-on: http://gerrit.cloudera.org:8080/17291
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
(cherry picked from commit 6aeb466c538d8cc566031c4a82200f18f088e18a)
Reviewed-on: http://gerrit.cloudera.org:8080/17293
---
M src/kudu/util/maintenance_manager-test.cc
M src/kudu/util/maintenance_manager.cc
2 files changed, 25 insertions(+), 2 deletions(-)

Approvals:
  Alexey Serbin: Looks good to me, approved
  Andrew Wong: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration

Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/17293 )

Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................


Patch Set 1: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 09 Apr 2021 00:37:43 +0000
Gerrit-HasComments: No

[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has removed a vote on this change.

Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR](branch-1.14.x) KUDU-3268: fix race between MM scheduling and unregistration

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/17293 )

Change subject: KUDU-3268: fix race between MM scheduling and unregistration
......................................................................


Patch Set 1: Verified+1

Test failures in RELEASE and DEBUG mode seem to be unrelated (python issues).


-- 
To view, visit http://gerrit.cloudera.org:8080/17293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: branch-1.14.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I29ad4d545de8166832b7c4148880a7e8719353bf
Gerrit-Change-Number: 17293
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 10 Apr 2021 00:03:04 +0000
Gerrit-HasComments: No