You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2019/06/28 20:58:09 UTC
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13763
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of kudu CLI process, it seems safe to mute it for a while.
The motivation for this change is to hunt down another issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/13763/1
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13763
to look at the new patch set (#5).
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of the kudu CLI tool, it seems safe to mute it.
The motivation for this change is to hunt down the issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
The suppression of the warnings for a locked mutex relies on the fix for
issue 944 in the thread sanitizers. The fix is back-ported as a patch
for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
fix, see https://github.com/google/sanitizers/issues/944.
The results of two representative 1K runs are below: kudu-admin-test built
in TSAN with --gtest_filter='*ToolTest.TestDeleteTable*' before and after
the fix (yes, I ran many of 1K runs for both versions).
before: 1 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931838.118516
after : 0 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931149.115149
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 3 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/13763/5
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13763 )
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of the kudu CLI tool, it seems safe to mute it.
The motivation for this change is to hunt down the issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
The suppression of the warnings for a locked mutex relies on the fix for
issue 944 in the thread sanitizers. The fix is back-ported as a patch
for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
fix, see https://github.com/google/sanitizers/issues/944.
The results of two representative 1K runs are below: kudu-admin-test built
in TSAN with --gtest_filter='*ToolTest.TestDeleteTable*' before and after
the fix (yes, I ran many of 1K runs for both versions).
before: 1 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931838.118516
after : 0 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931149.115149
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Reviewed-on: http://gerrit.cloudera.org:8080/13763
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 3 insertions(+), 0 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13763 )
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13763/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13763/5//COMMIT_MSG@17
PS5, Line 17:
: The suppression of the warnings for a locked mutex relies on the fix for
: issue 944 in the thread sanitizers. The fix is back-ported as a patch
: for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
: fix, see https://github.com/google/sanitizers/issues/944.
> Does the stack trace of the TSAN warning change with the application of tha
The stack traces for TSAN warnings look pretty much the same if not suppressing the warning (i.e. I don't see variations as reported in issue 944).
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc
File src/kudu/util/sanitizer_options.cc:
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc@113
PS5, Line 113: // See https://issues.apache.org/jira/browse/KUDU-2212 for details on
: // the destruction of a locked mutex in glog.
> Could you make sure this is reflected to upstream glog too? I'm not sure if
Yep -- I'll add the related piece into issue #80 reported by Todd.
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 17 Aug 2019 00:10:00 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13763 )
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
Patch Set 5: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13763/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13763/5//COMMIT_MSG@17
PS5, Line 17:
: The suppression of the warnings for a locked mutex relies on the fix for
: issue 944 in the thread sanitizers. The fix is back-ported as a patch
: for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
: fix, see https://github.com/google/sanitizers/issues/944.
Does the stack trace of the TSAN warning change with the application of that LLVM backport?
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc
File src/kudu/util/sanitizer_options.cc:
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc@113
PS5, Line 113: // See https://issues.apache.org/jira/browse/KUDU-2212 for details on
: // the destruction of a locked mutex in glog.
Could you make sure this is reflected to upstream glog too? I'm not sure if it's captured in issue #80.
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 16 Aug 2019 19:13:57 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13763
to look at the new patch set (#3).
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of the kudu CLI tool, it seems safe to mute it.
The motivation for this change is to hunt down the issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
The suppression of the warnings for a locked mutex relies on the fix for
issue 944 in the thread sanitizers. The fix is back-ported as a patch
for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
fix, see https://github.com/google/sanitizers/issues/944.
The results of two representative 1K runs are below: kudu-admin-test built
in TSAN with --gtest_filter='*ToolTest.TestDeleteTable*' before and after
the fix (yes, I ran many of 1K runs for both versions).
before: 1 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931838.118516
after : 0 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931149.115149
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/13763/3
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13763
to look at the new patch set (#2).
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of kudu CLI process, it seems safe to mute it for a while. In addition,
I added a suppression for a warning about possible deadlock while
shutting down libglog's runtime.
The motivation for this change is to hunt down another issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
The suppression of the warnings for a locked mutex relies on the fix for
issue 944 in the thread sanitizers. The fix is back-ported as a patch
for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
fix, see https://github.com/google/sanitizers/issues/944.
The results of two representative 1K runs are below: kudu-admin-test built
in TSAN with --gtest_filter='*ToolTest.TestDeleteTable*' before and after
the fix.
before 2 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565930436.112333
after 0 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931149.115149
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/13763/2
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13763 )
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 28 Jun 2019 23:37:53 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13763 )
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc
File src/kudu/util/sanitizer_options.cc:
http://gerrit.cloudera.org:8080/#/c/13763/5/src/kudu/util/sanitizer_options.cc@113
PS5, Line 113: // See https://issues.apache.org/jira/browse/KUDU-2212 for details on
: // the destruction of a locked mutex in glog.
> Yep -- I'll add the related piece into issue #80 reported by Todd.
After trying to put together a concise description of the issue for upstream glog, I realized this is not about self-contained glog issue. Rather, it's about using glog functionality (i.e. sending log messages) while finalizing/unloading the glog library.
Let me know if you think otherwise.
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 17 Aug 2019 00:47:44 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2212 mute TSAN on destruction of a locked mutex in glog
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13763
to look at the new patch set (#4).
Change subject: KUDU-2212 mute TSAN on destruction of a locked mutex in glog
......................................................................
KUDU-2212 mute TSAN on destruction of a locked mutex in glog
This patch adds a suppression for TSAN warning about destruction
of a locked mutex in libglog. Since the only registered instances
of the issue happen only during shutdown of the C++ runtime upon exit
of the kudu CLI tool, it seems safe to mute it.
The motivation for this change is to hunt down the issue tracked
by KUDU-2878: while doing so, this benign TSAN warning hinders the
scenario used to RCA the latter.
The suppression of the warnings for a locked mutex relies on the fix for
issue 944 in the thread sanitizers. The fix is back-ported as a patch
for LLVM 6.0 in Kudu's thirdparty. For details on the TSAN suppressions
fix, see https://github.com/google/sanitizers/issues/944.
The results of two representative 1K runs are below: kudu-admin-test built
in TSAN with --gtest_filter='*ToolTest.TestDeleteTable*' before and after
the fix (yes, I ran many of 1K runs for both versions).
before: 1 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931838.118516
after : 0 out of 1K failed:
http://dist-test.cloudera.org/job?job_id=aserbin.1565931149.115149
Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
---
M src/kudu/util/sanitizer_options.cc
1 file changed, 5 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/13763/4
--
To view, visit http://gerrit.cloudera.org:8080/13763
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If0b62027a1a4db18bfaa6c89b3070c918fc4f88a
Gerrit-Change-Number: 13763
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)