[jira] [Commented] (IMPALA-11370) Run make with --load-average for large toolchain components

["Joe McDonnell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572563#comment-17572563 ] 

Joe McDonnell commented on IMPALA-11370:
----------------------------------------

https://github.com/cloudera/native-toolchain/commit/b2aaa135834ecfdafa146da64ccaae023fdbd27a
{noformat}
IMPALA-11370: Run make with --load-average for GCC, LLVM, and Kudu
Large C++ projects can saturate large machines. When
the docker build infrastructures runs multiple builds
in parallel, it can overload the machine and cause
high memory usage / instability.

This adds "--load-average" to the make commands for three
large projects: GCC, LLVM, and Kudu. This can be tuned
later to get higher utilization.

Change-Id: I621e3f994d0266557e0a3d93ce329e36a90b7ae6
Reviewed-on: http://gerrit.cloudera.org:8080/18655
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Joe McDonnell <jo...@cloudera.com>
{noformat}

> Run make with --load-average for large toolchain components
> -----------------------------------------------------------
>
>                 Key: IMPALA-11370
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11370
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Infrastructure
>    Affects Versions: Impala 4.2.0
>            Reporter: Joe McDonnell
>            Priority: Major
>
> Several components of the toolchain can use a significant amount of parallelism and resources (e.g. LLVM, Kudu). When using the docker build infrastructures, multiple of these builds can happen simultaneously. This can result in overloading the machine and high memory usage / memory pressure.
> One way to deal with it is to run those builds with make's --load-average command so that the total load doesn't go too high. This can result in a sawtooth load pattern as the make commands cut off and the load drifts down, so this may need some tuning of the value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org