You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2025/11/05 19:41:20 UTC
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23638
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
IMPALA-14530: Limit link concurrency with flock
Adds a wrapper around 'gold' that limits concurrency using 'flock'.
Concurrency is limited to IMPALA_LINK_THREADS by using a pool of link
files and randomly selecting from them.
Uses 'ld.gold' to select the version in binutils, and '--threads' to use
more cores while still limiting memory from multiple links.
New link option is selected by setting IMPALA_LINKER=fold, which is now
set as the default.
Adds MOLD_JOBS=1 to avoid overloading the system when trying 'mold' and
linking test binaries. However 'mold' is not selected as the default
due to test failures around SASL/GSSAPI.
Removes the guard in bootstrap_development.sh limiting
IMPALA_BUILD_THREADS as it's no longer needed with default config.
Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
---
M .gitignore
M README-build.md
M be/CMakeLists.txt
M bin/bootstrap_development.sh
A bin/fold/ld
M bin/impala-config.sh
6 files changed, 49 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23638/1
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-13147: Limit link concurrency with flock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-13147: Limit link concurrency with flock
......................................................................
Patch Set 3:
> Patch Set 3:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/20833/ : Initial code review checks failed. See linked job for details on the failure.
Besides issue with using ninja, build-only failed on Python 2. Seems like a transient error, but it'd be nice to get rid of Python 2.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 16:56:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23638
to look at the new patch set (#2).
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
IMPALA-14530: Limit link concurrency with flock
Adds a wrapper around 'gold' that limits concurrency using 'flock'.
Concurrency is limited to IMPALA_LINK_THREADS by using a pool of link
files and randomly selecting from them.
Uses 'ld.gold' to select the version in binutils, and '--threads' to use
more cores while still limiting memory from multiple links.
New link option is selected by setting IMPALA_LINKER=fold, which is now
set as the default. Performance does not appear to be significantly
worse than before, and link actually succeeds on rebuild dominated by
linking test binaries.
Adds MOLD_JOBS=1 to avoid overloading the system when trying 'mold' and
linking test binaries. However 'mold' is not selected as the default
due to test failures around SASL/GSSAPI.
Removes the guard in bootstrap_development.sh limiting
IMPALA_BUILD_THREADS as it's no longer needed with default config.
Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
---
M .gitignore
M README-build.md
M be/CMakeLists.txt
M bin/bootstrap_development.sh
A bin/fold/ld
M bin/impala-config.sh
6 files changed, 56 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23638/2
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13147: Limit link concurrency with flock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23638
to look at the new patch set (#3).
Change subject: IMPALA-13147: Limit link concurrency with flock
......................................................................
IMPALA-13147: Limit link concurrency with flock
Adds a wrapper around 'gold' that limits concurrency using 'flock'.
Concurrency is limited to IMPALA_LINK_THREADS by using a pool of link
files and randomly selecting from them.
Uses 'ld.gold' to select the version in binutils, and '--threads' to use
more cores while still limiting memory from multiple links.
New link option is selected by setting IMPALA_LINKER=fold. Performance
does not appear to be significantly worse than before, and link actually
succeeds on rebuild dominated by linking test binaries with 'make'.
Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
---
M .gitignore
M README-build.md
M be/CMakeLists.txt
A bin/fold/ld
M bin/impala-config.sh
5 files changed, 42 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/23638/3
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has restored this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
Restored
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: restore
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13147: Limit link concurrency with flock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-13147: Limit link concurrency with flock
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20833/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 01:16:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has abandoned this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
Abandoned
I'm going to combine this with https://gerrit.cloudera.org/c/23572/. All still somewhat experimental.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13147: Limit link concurrency with flock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-13147: Limit link concurrency with flock
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/20842/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Nov 2025 20:08:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20827/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Nov 2025 21:12:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14530: Limit link concurrency with flock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-14530: Limit link concurrency with flock
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/20825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Nov 2025 20:07:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13147: Limit link concurrency with flock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/23638 )
Change subject: IMPALA-13147: Limit link concurrency with flock
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23638/4/bin/fold/ld
File bin/fold/ld:
http://gerrit.cloudera.org:8080/#/c/23638/4/bin/fold/ld@24
PS4, Line 24: $IMPALA_HOME/.linklock$POOL
Nit: Would be better to surround with double quotes and curly braces:
"${IMPALA_HOME}/.linklock${POOL}"
--
To view, visit http://gerrit.cloudera.org:8080/23638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9b1be517bbc439a17932cc16c8a203e3bc4e5e04
Gerrit-Change-Number: 23638
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 18 Nov 2025 22:15:02 +0000
Gerrit-HasComments: Yes