You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/05/15 02:23:02 UTC

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22900


Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................

IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

IMPALA-13978 switched HdrHistrogram from using a gscoped_ptr
to unique_ptr. This has been causing TSAN issues during the
teardown for expr-test. gscoped_ptr doesn't null out the
pointer when it gets destructed, but unique_ptr does. This
is a data race with the threads that are still running and
trying to access the metrics.

The full solution would be to have an orderly shutdown of
all the threads before destructing things. That is a large
project that would touch many different components. As a
short-term fix, this avoids the TSAN issue by leaking
the statestore metrics.

We should consider fixing IMPALA-9314 and implementing
orderly shutdown.

Testing:
 - Ran expr-test in a loop with TSAN and didn't see this
   particular issue. There are other shutdown issues with
   much lower frequency that have different symptoms.

Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
---
M be/src/exprs/expr-test.cc
1 file changed, 9 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/22900/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1: Code-Review+2

(1 comment)

I think this is OK in test code.

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc@230
PS1, Line 230: IGNORE_LEAKING_OBJECT(statestore_metrics);
> Yes, the object will live for the lifetime of the process and not get freed
Thanks for the explanation.



-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 04:02:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11885/


-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 11:38:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc@230
PS1, Line 230: IGNORE_LEAKING_OBJECT(statestore_metrics);
> So this macro just tell thread sanitizer to ignore leak in this spot?
Yes, the object will live for the lifetime of the process and not get freed before exit, so this lets the sanitizers know that. I think it may not actually impact anything at the moment, but we do that with other leaked objects here (like statestore and impala_server).



-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 03:59:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11888/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 15:19:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 21:38:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19217/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 02:49:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc
File be/src/exprs/expr-test.cc:

http://gerrit.cloudera.org:8080/#/c/22900/1/be/src/exprs/expr-test.cc@230
PS1, Line 230: IGNORE_LEAKING_OBJECT(statestore_metrics);
So this macro just tell thread sanitizer to ignore leak in this spot?



-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 02:28:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11885/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 May 2025 05:17:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22900 )

Change subject: IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test
......................................................................

IMPALA-14049: Fix TSAN issue with HdrHistogram in expr-test

IMPALA-13978 switched HdrHistrogram from using a gscoped_ptr
to unique_ptr. This has been causing TSAN issues during the
teardown for expr-test. gscoped_ptr doesn't null out the
pointer when it gets destructed, but unique_ptr does. This
is a data race with the threads that are still running and
trying to access the metrics.

The full solution would be to have an orderly shutdown of
all the threads before destructing things. That is a large
project that would touch many different components. As a
short-term fix, this avoids the TSAN issue by leaking
the statestore metrics.

We should consider fixing IMPALA-9314 and implementing
orderly shutdown.

Testing:
 - Ran expr-test in a loop with TSAN and didn't see this
   particular issue. There are other shutdown issues with
   much lower frequency that have different symptoms.

Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Reviewed-on: http://gerrit.cloudera.org:8080/22900
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exprs/expr-test.cc
1 file changed, 9 insertions(+), 2 deletions(-)

Approvals:
  Riza Suminto: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/22900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I73c3f4db16c6ffa272f2512e9871db5743be7a54
Gerrit-Change-Number: 22900
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>