You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2025/12/13 09:40:46 UTC
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23785
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
KUDU-3545 fix race condition when (un)registring EH frames
This patch addresses race condition in libgcc which is triggered by
concurrent registering/unregistering of EH frames by LLVM's
RTDyldMemoryManager. Also, since libgcc expects a null-terminated
list of FDEs, this patch adds extra 4 zero bytes in the end of each
section allocated for EH frames.
I added a new test scenario to reproduce the race, and it was triggering
TSAN warnings every time if rolling back the fix (see below). For more
robust and more reliable race reproduction, I added a new flag
(tagged as 'hidden', default value is 1) to control the maximum number
of threads in the codegen compiler thread pool:
--codegen_compiler_manager_pool_max_threads_num
We might start using it later to allow for parallel compilation of row
projection functions in the presence of concurrent scan requests.
WARNING: ThreadSanitizer: data race (pid=3406438)
Write of size 8 at 0x7b4000008228 by thread T6 (mutexes: write M1879):
#0 memmove sanitizer_common/sanitizer_common_interceptors.inc:791:3 (codegen-test+0x2edb86)
#1 <null> <null> (libgcc_s.so.1+0x1345d)
#2 llvm::RuntimeDyldELF::registerEHFrames()
#3 llvm::RuntimeDyld::registerEHFrames()
#4 llvm::MCJIT::finalizeLoadedModules()
#5 llvm::MCJIT::finalizeObject()
#6 kudu::codegen::ModuleBuilder::Compile()
...
Previous write of size 8 at 0x7b4000008228 by thread T5 (mutexes: write M1869):
#0 malloc /root/Projects/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:652:5 (codegen-test+0x2b2c34)
#1 <null> <null> (libgcc_s.so.1+0x1318a)
#2 llvm::RuntimeDyldELF::registerEHFrames()
#3 llvm::RuntimeDyld::registerEHFrames()
#4 llvm::MCJIT::finalizeLoadedModules()
#5 llvm::MCJIT::finalizeObject()
#6 kudu::codegen::ModuleBuilder::Compile()
#7 kudu::codegen::RowProjectorFunctions::Create()
#8 kudu::codegen::CodeGenerator::CompileRowProjector()
...
SUMMARY: ThreadSanitizer: data race (/lib64/libgcc_s.so.1+0x1345d)
Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
---
M src/kudu/codegen/CMakeLists.txt
M src/kudu/codegen/codegen-test.cc
M src/kudu/codegen/compilation_manager.cc
A src/kudu/codegen/jit_frame_manager.cc
A src/kudu/codegen/jit_frame_manager.h
M src/kudu/codegen/module_builder.cc
6 files changed, 213 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/85/23785/1
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Zoltan Martonka (Code Review)" <ge...@cloudera.org>.
Zoltan Martonka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Dec 2025 11:01:37 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Abhishek Chennaka (Code Review)" <ge...@cloudera.org>.
Abhishek Chennaka has posted comments on this change. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Dec 2025 17:44:24 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has posted comments on this change. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Patch Set 1: Verified+1 Code-Review+2
(1 comment)
Thank you Alexey and Zoltan for digging into it and fixing it so quickly, LGTM.
Verified -1 looked like a flaky test failure, overriding it.
http://gerrit.cloudera.org:8080/#/c/23785/1/src/kudu/codegen/codegen-test.cc
File src/kudu/codegen/codegen-test.cc:
http://gerrit.cloudera.org:8080/#/c/23785/1/src/kudu/codegen/codegen-test.cc@457
PS1, Line 457: accross
nit: across
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Dec 2025 11:13:28 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has removed a vote on this change.
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Attila Bukor (Code Review)" <ge...@cloudera.org>.
Attila Bukor has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
KUDU-3545 fix race condition when (un)registring EH frames
This patch addresses race condition in libgcc which is triggered by
concurrent registering/unregistering of EH frames by LLVM's
RTDyldMemoryManager. Also, since libgcc expects a null-terminated
list of FDEs, this patch adds extra 4 zero bytes in the end of each
section allocated for EH frames.
I added a new test scenario to reproduce the race, and it was triggering
TSAN warnings every time if rolling back the fix (see below). For more
robust and more reliable race reproduction, I added a new flag
(tagged as 'hidden', default value is 1) to control the maximum number
of threads in the codegen compiler thread pool:
--codegen_compiler_manager_pool_max_threads_num
We might start using it later to allow for parallel compilation of row
projection functions in the presence of concurrent scan requests.
WARNING: ThreadSanitizer: data race (pid=3406438)
Write of size 8 at 0x7b4000008228 by thread T6 (mutexes: write M1879):
#0 memmove sanitizer_common/sanitizer_common_interceptors.inc:791:3 (codegen-test+0x2edb86)
#1 <null> <null> (libgcc_s.so.1+0x1345d)
#2 llvm::RuntimeDyldELF::registerEHFrames()
#3 llvm::RuntimeDyld::registerEHFrames()
#4 llvm::MCJIT::finalizeLoadedModules()
#5 llvm::MCJIT::finalizeObject()
#6 kudu::codegen::ModuleBuilder::Compile()
...
Previous write of size 8 at 0x7b4000008228 by thread T5 (mutexes: write M1869):
#0 malloc /root/Projects/kudu/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:652:5 (codegen-test+0x2b2c34)
#1 <null> <null> (libgcc_s.so.1+0x1318a)
#2 llvm::RuntimeDyldELF::registerEHFrames()
#3 llvm::RuntimeDyld::registerEHFrames()
#4 llvm::MCJIT::finalizeLoadedModules()
#5 llvm::MCJIT::finalizeObject()
#6 kudu::codegen::ModuleBuilder::Compile()
#7 kudu::codegen::RowProjectorFunctions::Create()
#8 kudu::codegen::CodeGenerator::CompileRowProjector()
...
SUMMARY: ThreadSanitizer: data race (/lib64/libgcc_s.so.1+0x1345d)
Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Reviewed-on: http://gerrit.cloudera.org:8080/23785
Reviewed-by: Zoltan Martonka <zm...@cloudera.com>
Reviewed-by: Attila Bukor <ab...@apache.org>
Tested-by: Attila Bukor <ab...@apache.org>
---
M src/kudu/codegen/CMakeLists.txt
M src/kudu/codegen/codegen-test.cc
M src/kudu/codegen/compilation_manager.cc
A src/kudu/codegen/jit_frame_manager.cc
A src/kudu/codegen/jit_frame_manager.h
M src/kudu/codegen/module_builder.cc
6 files changed, 213 insertions(+), 3 deletions(-)
Approvals:
Zoltan Martonka: Looks good to me, but someone else must approve
Attila Bukor: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Patch Set 1: Verified-1
Build Failed
http://jenkins.kudu.apache.org/job/pre_commit/2187/ : FAILURE
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Ashwani Raina <ar...@gmail.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>
Gerrit-Reviewer: Zoltan Martonka <zm...@cloudera.com>
Gerrit-Comment-Date: Sat, 13 Dec 2025 10:35:01 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3545 fix race condition when (un)registring EH frames
Posted by "Kudu Jenkins (Code Review)" <ge...@cloudera.org>.
Kudu Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23785 )
Change subject: KUDU-3545 fix race condition when (un)registring EH frames
......................................................................
Patch Set 1:
Build Started http://jenkins.kudu.apache.org/job/pre_commit/2187/
--
To view, visit http://gerrit.cloudera.org:8080/23785
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia91ada71f663451e11149a190def6a8ccd0b4ef5
Gerrit-Change-Number: 23785
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 13 Dec 2025 09:40:52 +0000
Gerrit-HasComments: No