You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2017/09/06 18:12:11 UTC
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Tim Armstrong has uploaded a new change for review.
http://gerrit.cloudera.org:8080/7977
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
IMPALA-5902: add ThreadSanitizer build
This is sufficient to get Impala to come up and run queries with
thread sanitizer enabled.
I have not triaged or fixed the data races that are reported, that
is left for follow-on work.
Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/common/init.cc
M be/src/gutil/atomicops-internals-tsan.h
M be/src/runtime/bufferpool/system-allocator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-exec-mgr.cc
M be/src/util/default-path-handlers.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/metrics-test.cc
M be/src/util/pprof-path-handlers.cc
M bin/make_impala.sh
M bin/run-backend-tests.sh
M bin/start-catalogd.sh
M bin/start-impalad.sh
M bin/start-statestored.sh
M buildall.sh
M tests/common/environ.py
19 files changed, 91 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/7977/1
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1203/
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 1:
(8 comments)
Nice, thanks for taking this on.
Can you file a ticket to add this to https://jenkins.impala.io/job/all-build-options/ when this is done?
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/common/init.cc
File be/src/common/init.cc:
Line 140
No longer needed? Why not?
Line 199: #ifndef THREAD_SANITIZER
Why can't we run this with TSAN? Should the #ifndef go inside that function itself?
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/runtime/query-exec-mgr.cc
File be/src/runtime/query-exec-mgr.cc:
Line 115: // tcmalloc and address sanitizer cannot be used together
nit: "or thread"
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
Line 139: ss << "Memory tracking is not available with address sanitizer builds.";
nit: "or thread"
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/memory-metrics.cc
File be/src/util/memory-metrics.cc:
PS1, Line 86: asan
naming?
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/memory-metrics.h
File be/src/util/memory-metrics.h:
Line 136: /// Alternative to TCMallocMetric if we're running under Address Sanitizer, which
Naming?
http://gerrit.cloudera.org:8080/#/c/7977/1/bin/start-impalad.sh
File bin/start-impalad.sh:
PS1, Line 104: 7
How did you pick 7?
http://gerrit.cloudera.org:8080/#/c/7977/1/tests/common/environ.py
File tests/common/environ.py:
Line 135: either compiled with code coverage enabled or a sanitizer.
except ubsan
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.
Jim Apple has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/7977/1/tests/common/environ.py
File tests/common/environ.py:
Line 135: either compiled with code coverage enabled, ASAN or TSAN.
> I wasn't sure if UBSAN was just missing from this file - is it just treated
I suspect so.
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
IMPALA-5902: add ThreadSanitizer build
This is sufficient to get Impala to come up and run queries with
thread sanitizer enabled.
I have not triaged or fixed the data races that are reported, that
is left for follow-on work.
Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Reviewed-on: http://gerrit.cloudera.org:8080/7977
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/common/init.cc
M be/src/gutil/atomicops-internals-tsan.h
M be/src/runtime/bufferpool/system-allocator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-exec-mgr.cc
M be/src/util/default-path-handlers.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/metrics-test.cc
M be/src/util/pprof-path-handlers.cc
M bin/make_impala.sh
M bin/run-backend-tests.sh
M bin/start-catalogd.sh
M bin/start-impalad.sh
M bin/start-statestored.sh
M buildall.sh
M common/thrift/metrics.json
M tests/common/environ.py
20 files changed, 105 insertions(+), 52 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Tim Armstrong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 1:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/common/init.cc
File be/src/common/init.cc:
Line 142: // The metric is refreshed whenever memory is consumed or released via a MemTracker,
A while back I changed the Process MemTracker under ASAN to use a metric. This makes it reasonable to call this.
Line 199: #ifndef THREAD_SANITIZER
> Why can't we run this with TSAN? Should the #ifndef go inside that function
There are alternative implementations of the atomic ops for TSAN. That function is not even compiled under TSAN.
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/runtime/query-exec-mgr.cc
File be/src/runtime/query-exec-mgr.cc:
Line 115: // tcmalloc and address sanitizer cannot be used together
> nit: "or thread"
Done
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/default-path-handlers.cc
File be/src/util/default-path-handlers.cc:
Line 139: ss << "Memory tracking is not available with address sanitizer builds.";
> nit: "or thread"
Done
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/memory-metrics.cc
File be/src/util/memory-metrics.cc:
PS1, Line 86: asan
> naming?
Done
http://gerrit.cloudera.org:8080/#/c/7977/1/be/src/util/memory-metrics.h
File be/src/util/memory-metrics.h:
Line 136: /// Alternative to TCMallocMetric if we're running under Address Sanitizer, which
> Naming?
Done
http://gerrit.cloudera.org:8080/#/c/7977/1/bin/start-impalad.sh
File bin/start-impalad.sh:
PS1, Line 104: 7
> How did you pick 7?
7 is the maximum value. I saw a few instances where the errors didn't include all info because it was aged otu of the history. https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
I guess I should use the same for the other daemons too for consistency.
http://gerrit.cloudera.org:8080/#/c/7977/1/tests/common/environ.py
File tests/common/environ.py:
Line 135: either compiled with code coverage enabled or a sanitizer.
> except ubsan
I wasn't sure if UBSAN was just missing from this file - is it just treated as a normal debug build?
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 1:
Filed IMPALA-5905
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has uploaded a new patch set (#2).
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
IMPALA-5902: add ThreadSanitizer build
This is sufficient to get Impala to come up and run queries with
thread sanitizer enabled.
I have not triaged or fixed the data races that are reported, that
is left for follow-on work.
Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/common/init.cc
M be/src/gutil/atomicops-internals-tsan.h
M be/src/runtime/bufferpool/system-allocator.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/query-exec-mgr.cc
M be/src/util/default-path-handlers.cc
M be/src/util/memory-metrics.cc
M be/src/util/memory-metrics.h
M be/src/util/metrics-test.cc
M be/src/util/pprof-path-handlers.cc
M bin/make_impala.sh
M bin/run-backend-tests.sh
M bin/start-catalogd.sh
M bin/start-impalad.sh
M bin/start-statestored.sh
M buildall.sh
M common/thrift/metrics.json
M tests/common/environ.py
20 files changed, 105 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/7977/2
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-5902: add ThreadSanitizer build
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-5902: add ThreadSanitizer build
......................................................................
Patch Set 3: Code-Review+2
rebase
--
To view, visit http://gerrit.cloudera.org:8080/7977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I22f8faeefa5e157279c5973fe28bc573b7606d50
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No