[jira] [Assigned] (IMPALA-12939) Improve default for IMPALA_BUILD_THREADS

["Michael Smith (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Smith reassigned IMPALA-12939:
--------------------------------------

    Assignee: Michael Smith

> Improve default for IMPALA_BUILD_THREADS
> ----------------------------------------
>
>                 Key: IMPALA-12939
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12939
>             Project: IMPALA
>          Issue Type: Task
>          Components: Infrastructure
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Major
>
> Improve the default selection for IMPALA_BUILD_THREADS and other parallelization config.
> Impala's build process needs 2GB of memory per CPU core with {{-notests}} or 4GB of memory per CPU core when building unit test binaries (the link process is especially memory-intensive). Exceeding these can lead to systems with many cores running out of memory during the build. We currently do not consider memory when selecting a default value for IMPALA_BUILD_THREADS.
> We currently default IMPALA_BUILD_THREADS to {{{}nproc{}}}, which also does not reflect CPU slicing that may happen in containers. How to detect this differs between cgroups v1 and v2 (v2 is becoming more common, present on Ubuntu 22 and RedHat 9).
>  * v1:
> {code:java}
> echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us))){code}
>  * v2:
> {code:java}
> awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max{code}
> Something like the following should handle all cases
> {code}
> if [[ -f /sys/fs/cgroup/cpu.max ]]; then
>   awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max
> elif [[ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]]; then
>   echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(/sys/fs/cgroup/cpu/cpu.cfs_period_us)))
> else
>   nproc
> fi
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org