You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/04/30 00:36:05 UTC

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22834


Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................

IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Upstream gperftools does not allow setting tcmalloc.max_total_thread_cache_bytes
to greater than 1GB. This moves to a new toolchain that has patched
gperftools to remove this limitation and allow setting
tcmalloc.max_total_thread_cache_bytes > 1GB. This also reads back the
value from tcmalloc and prints a warning if it doesn't match what we set.

Testing:
 - Set tcmalloc_max_total_thread_cache_bytes to 2GB and verified that
   the warning message doesn't appear. On unpatched versions of
   gperftools, the warning message does appear.

Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
---
M be/src/runtime/exec-env.cc
M bin/impala-config.sh
2 files changed, 24 insertions(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/22834/1
-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................

IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Upstream gperftools does not allow setting tcmalloc.max_total_thread_cache_bytes
to greater than 1GB. This moves to a new toolchain that has patched
gperftools to remove this limitation and allow setting
tcmalloc.max_total_thread_cache_bytes > 1GB. This also reads back the
value from tcmalloc and prints a warning if it doesn't match what we set.

Testing:
 - Set tcmalloc_max_total_thread_cache_bytes to 2GB and verified that
   the warning message doesn't appear. On unpatched versions of
   gperftools, the warning message does appear.

Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Reviewed-on: http://gerrit.cloudera.org:8080/22834
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
---
M be/src/runtime/exec-env.cc
M bin/impala-config.sh
2 files changed, 25 insertions(+), 6 deletions(-)

Approvals:
  Michael Smith: Looks good to me, but someone else must approve
  Impala Public Jenkins: Verified
  Joe McDonnell: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 4
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 2: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 00:38:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 01 May 2025 03:37:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11816/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 01:43:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3:

Clang was failing on ASAN with "unused variable", so moved the constant down to the function body.


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 17:14:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 17:38:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19058/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 01:05:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 1:

(1 comment)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/22834/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:

http://gerrit.cloudera.org:8080/#/c/22834/1/be/src/runtime/exec-env.cc@503
PS1, Line 503:   if (actual_max_total_thread_cache_bytes != FLAGS_tcmalloc_max_total_thread_cache_bytes) {
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 00:37:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 1: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 00:38:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/22834/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:

http://gerrit.cloudera.org:8080/#/c/22834/1/be/src/runtime/exec-env.cc@503
PS1, Line 503:   if (actual_max_total_thread_cache_bytes != FLAGS_tcmalloc_max_total_thread_cache_bytes) {
> line too long (91 > 90)
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 00:38:35 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 01:02:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 2: Code-Review+2

Carry Michael's +1


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 01:17:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11823/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 21:27:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3: Code-Review+2

I'm going to go ahead with this, since this had +2 before and the last change was trivial.


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 01 May 2025 16:41:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11816/


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 07:49:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/22834

to look at the new patch set (#2).

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................

IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Upstream gperftools does not allow setting tcmalloc.max_total_thread_cache_bytes
to greater than 1GB. This moves to a new toolchain that has patched
gperftools to remove this limitation and allow setting
tcmalloc.max_total_thread_cache_bytes > 1GB. This also reads back the
value from tcmalloc and prints a warning if it doesn't match what we set.

Testing:
 - Set tcmalloc_max_total_thread_cache_bytes to 2GB and verified that
   the warning message doesn't appear. On unpatched versions of
   gperftools, the warning message does appear.

Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
---
M be/src/runtime/exec-env.cc
M bin/impala-config.sh
2 files changed, 25 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/22834/2
-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/22834

to look at the new patch set (#3).

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................

IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Upstream gperftools does not allow setting tcmalloc.max_total_thread_cache_bytes
to greater than 1GB. This moves to a new toolchain that has patched
gperftools to remove this limitation and allow setting
tcmalloc.max_total_thread_cache_bytes > 1GB. This also reads back the
value from tcmalloc and prints a warning if it doesn't match what we set.

Testing:
 - Set tcmalloc_max_total_thread_cache_bytes to 2GB and verified that
   the warning message doesn't appear. On unpatched versions of
   gperftools, the warning message does appear.

Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
---
M be/src/runtime/exec-env.cc
M bin/impala-config.sh
2 files changed, 25 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/22834/3
-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/22834 )

Change subject: IMPALA-13479: Patch gperftools to remove 1GB limit on thread caches
......................................................................


Patch Set 3: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/22834
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If78c8734c704090c12737a8c2a8456b73ea4b8e8
Gerrit-Change-Number: 22834
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Apr 2025 17:23:29 +0000
Gerrit-HasComments: No