You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2018/02/28 07:47:13 UTC
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Hello Will Berkeley, Mike Percy, Alexey Serbin,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/9460
to review the following change.
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
The wrapping of libdl functions added by commit
d9e7037138646e3efb331af98c6982de13294c4b has a problem if any other
dynamic initializer calls dlopen or dlclose. It turns out that OpenSSL
in FIPS mode does indeed do that, leading to a crash with a stack like:
#0 0x0000000000000000 in ?? ()
#1 0x0000000001b45d23 in dlopen ()
#2 0x00007f1f444967ba in ?? () from /lib64/libcrypto.so.1.0.0
#3 0x00007f1f44496857 in ?? () from /lib64/libcrypto.so.1.0.0
#4 0x00007f1f44496bfe in FIPS_module_mode_set () from /lib64/libcrypto.so.1.0.0
#5 0x00007f1f4437216c in FIPS_mode_set () from /lib64/libcrypto.so.1.0.0
#6 0x00007f1f4436eb60 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
#7 0x00007f1f450a2c0a in call_init.part () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f1f450a2cf3 in _dl_init () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f1f4509518a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
The fix takes the same approach we already used to workaround a similar issue
with the ASAN runtime, but generalizes it to all of our wrapped functions.
Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
---
M src/kudu/util/debug/unwind_safeness.cc
1 file changed, 27 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/9460/1
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc
File src/kudu/util/debug/unwind_safeness.cc:
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc@89
PS1, Line 89: __attribute__((constructor))
: void InitIfNecessary() {
: // Dynamic library initialization is always single-threaded, so there's no
: // need for any synchronization here.
: if (g_initted) return;
:
: g_orig_dlopen = dlsym_or_die("dlopen");
: g_orig_dlclose = dlsym_or_die("dlclose");
: #ifndef __APPLE__ // This function doesn't exist on macOS.
: g_orig_dl_iterate_phdr = dlsym_or_die("dl_iterate_phdr");
: #endif
: g_initted = true;
: }
> Per the comment above, this ensures that the first time it is called is def
Ah, sure. I was under impression that now we always call some dlopen () before main(). But it would be safer not to assume that, of course.
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Feb 2018 09:08:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Mike Percy, Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9460
to look at the new patch set (#2).
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
The wrapping of libdl functions added by commit
d9e7037138646e3efb331af98c6982de13294c4b has a problem if any other
dynamic initializer calls dlopen or dlclose. It turns out that OpenSSL
in FIPS mode does indeed do that, leading to a crash with a stack like:
#0 0x0000000000000000 in ?? ()
#1 0x0000000001b45d23 in dlopen ()
#2 0x00007f1f444967ba in ?? () from /lib64/libcrypto.so.1.0.0
#3 0x00007f1f44496857 in ?? () from /lib64/libcrypto.so.1.0.0
#4 0x00007f1f44496bfe in FIPS_module_mode_set () from /lib64/libcrypto.so.1.0.0
#5 0x00007f1f4437216c in FIPS_mode_set () from /lib64/libcrypto.so.1.0.0
#6 0x00007f1f4436eb60 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
#7 0x00007f1f450a2c0a in call_init.part () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f1f450a2cf3 in _dl_init () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f1f4509518a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
The fix takes the same approach we already used to workaround a similar issue
with the ASAN runtime, but generalizes it to all of our wrapped functions.
Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
---
M src/kudu/util/debug/unwind_safeness.cc
1 file changed, 27 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/9460/2
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc
File src/kudu/util/debug/unwind_safeness.cc:
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc@89
PS1, Line 89: // Dynamic library initialization is always single-threaded, so there's no
: // need for any synchronization here.
: if (g_initted) return;
:
: g_orig_dlopen = dlsym_or_die("dlopen");
: g_orig_dlclose = dlsym_or_die("dlclose");
: #ifndef __APPLE__ // This function doesn't exist on macOS.
: g_orig_dl_iterate_phdr = dlsym_or_die("dl_iterate_phdr");
: #endif
: g_initted = true;
: }
:
: }
> If calling this from dlopen()/dlclose() anyway, why to have it as 'constru
Per the comment above, this ensures that the first time it is called is definitely during startup when things are single-threaded. If we waited for the first call, that could race against a "first call" from a second thread which we want to avoid.
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Feb 2018 08:47:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has removed a vote on this change.
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Patch Set 2: Verified+1
Unrelated flake. Also I verified this fixed the issue on sles12sp2 (was also able to verify the crash prior to the fix)
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Feb 2018 08:47:43 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
The wrapping of libdl functions added by commit
d9e7037138646e3efb331af98c6982de13294c4b has a problem if any other
dynamic initializer calls dlopen or dlclose. It turns out that OpenSSL
in FIPS mode does indeed do that, leading to a crash with a stack like:
#0 0x0000000000000000 in ?? ()
#1 0x0000000001b45d23 in dlopen ()
#2 0x00007f1f444967ba in ?? () from /lib64/libcrypto.so.1.0.0
#3 0x00007f1f44496857 in ?? () from /lib64/libcrypto.so.1.0.0
#4 0x00007f1f44496bfe in FIPS_module_mode_set () from /lib64/libcrypto.so.1.0.0
#5 0x00007f1f4437216c in FIPS_mode_set () from /lib64/libcrypto.so.1.0.0
#6 0x00007f1f4436eb60 in OPENSSL_init_library () from /lib64/libcrypto.so.1.0.0
#7 0x00007f1f450a2c0a in call_init.part () from /lib64/ld-linux-x86-64.so.2
#8 0x00007f1f450a2cf3 in _dl_init () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f1f4509518a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
The fix takes the same approach we already used to workaround a similar issue
with the ASAN runtime, but generalizes it to all of our wrapped functions.
Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Reviewed-on: http://gerrit.cloudera.org:8080/9460
Tested-by: Todd Lipcon <to...@apache.org>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
M src/kudu/util/debug/unwind_safeness.cc
1 file changed, 27 insertions(+), 11 deletions(-)
Approvals:
Todd Lipcon: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Feb 2018 09:08:40 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/9460 )
Change subject: KUDU-2328. Fix crash at startup with OpenSSL FIPS mode
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc
File src/kudu/util/debug/unwind_safeness.cc:
http://gerrit.cloudera.org:8080/#/c/9460/1/src/kudu/util/debug/unwind_safeness.cc@89
PS1, Line 89: __attribute__((constructor))
: void InitIfNecessary() {
: // Dynamic library initialization is always single-threaded, so there's no
: // need for any synchronization here.
: if (g_initted) return;
:
: g_orig_dlopen = dlsym_or_die("dlopen");
: g_orig_dlclose = dlsym_or_die("dlclose");
: #ifndef __APPLE__ // This function doesn't exist on macOS.
: g_orig_dl_iterate_phdr = dlsym_or_die("dl_iterate_phdr");
: #endif
: g_initted = true;
: }
If calling this from dlopen()/dlclose() anyway, why to have it as 'constructor'-like function to call before main at all?
--
To view, visit http://gerrit.cloudera.org:8080/9460
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10a04126411f51b4d8e290a6b061aa585aad0769
Gerrit-Change-Number: 9460
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 28 Feb 2018 08:34:40 +0000
Gerrit-HasComments: Yes