You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2024/03/25 22:33:33 UTC
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21200
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Also computes CORES from cgroups limits
if applicable. Uses IMPALA_BUILD_THREADS for load-data.
Removes NUM_CONCURRENT_TESTS from bootstrap_development.sh to avoid
overriding the default of # CPUs / 2, and the script doesn't run tests.
Also fixes issues that occur building in devcontainer:
* Adds default in case USER is unset during bootstrap.
* Adds a workaround for 'tar' error that occurs in Docker on MacOS
Cannot utime: No such file or directory
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
---
M bin/bootstrap_development.sh
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/load-data.py
5 files changed, 28 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21200/1
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/15666/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Mar 2024 01:41:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/15671/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Mar 2024 05:46:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21200
to look at the new patch set (#2).
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Also computes CORES from cgroups limits
if applicable. Uses IMPALA_BUILD_THREADS for load-data.
Removes NUM_CONCURRENT_TESTS from bootstrap_development.sh to avoid
overriding the default of # CPUs / 2, and the script doesn't run tests.
Also fixes issues that occur building in devcontainer:
* Adds default in case USER is unset during bootstrap.
* Adds a workaround for 'tar' error that occurs in Docker on MacOS
Cannot utime: No such file or directory
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
---
M bin/bootstrap_development.sh
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/load-data.py
5 files changed, 37 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21200/2
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 5: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Mar 2024 15:00:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21200
to look at the new patch set (#5).
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Also computes CORES from cgroups limits
if applicable. Uses IMPALA_BUILD_THREADS for load-data.
Removes NUM_CONCURRENT_TESTS from bootstrap_development.sh to avoid
overriding the default of # CPUs / 2, and the script doesn't run tests.
Also adds default in case USER is unset during bootstrap, which can
occur in devcontainer.
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
---
M bin/bootstrap_development.sh
M bin/bootstrap_system.sh
M bin/impala-config.sh
M bin/load-data.py
4 files changed, 54 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21200/5
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/15678/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Mar 2024 17:13:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21200
to look at the new patch set (#4).
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Also computes CORES from cgroups limits
if applicable. Uses IMPALA_BUILD_THREADS for load-data.
Removes NUM_CONCURRENT_TESTS from bootstrap_development.sh to avoid
overriding the default of # CPUs / 2, and the script doesn't run tests.
Also fixes issues that occur building in devcontainer:
* Adds default in case USER is unset during bootstrap.
* Adds a workaround for 'tar' error that occurs in Docker on MacOS
Cannot utime: No such file or directory
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
---
M bin/bootstrap_development.sh
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/load-data.py
5 files changed, 37 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21200/4
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/15673/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Mar 2024 04:49:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21200
to look at the new patch set (#3).
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
Updates IMPALA_BUILD_THREADS to bound it based on guideline of 2 GB
memory per core during builds. Also computes CORES from cgroups limits
if applicable. Uses IMPALA_BUILD_THREADS for load-data.
Removes NUM_CONCURRENT_TESTS from bootstrap_development.sh to avoid
overriding the default of # CPUs / 2, and the script doesn't run tests.
Also fixes issues that occur building in devcontainer:
* Adds default in case USER is unset during bootstrap.
* Adds a workaround for 'tar' error that occurs in Docker on MacOS
Cannot utime: No such file or directory
Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
---
M bin/bootstrap_development.sh
M bin/bootstrap_system.sh
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
M bin/load-data.py
5 files changed, 39 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/21200/3
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/bootstrap_system.sh@479
PS1, Line 479: echo -e "${USER-$(id -un)} - memlock 65536" | sudo tee /etc/security/limits.d/10-memlock.conf
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/impala-config.sh@909
PS1, Line 909: CORES=$(paste /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpu.cfs_period_us | awk "$awk_divide_roundup")
line too long (117 > 90)
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/load-data.py
File bin/load-data.py:
http://gerrit.cloudera.org:8080/#/c/21200/1/bin/load-data.py@79
PS1, Line 79: ]
flake8: E501 line too long (92 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 25 Mar 2024 22:34:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12939: Bound IMPALA BUILD THREADS for cgroups and memory
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21200 )
Change subject: IMPALA-12939: Bound IMPALA_BUILD_THREADS for cgroups and memory
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/15664/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/21200
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I87994d0464073fe2d91bc2f7c2592c012e42de71
Gerrit-Change-Number: 21200
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Mar 2024 00:03:08 +0000
Gerrit-HasComments: No