[jira] [Commented] (IMPALA-13147) Add support for limiting the concurrency of link jobs

["Michael Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18035673#comment-18035673 ] 

Michael Smith commented on IMPALA-13147:
----------------------------------------

I'm going to keep this open to explore ways to limit concurrency. I currently have two options: use ninja and pools, or use flock to limit concurrent links. I also explored making mold the default linker with MOLD_JOBS=1, but it currently has some sort of bug that makes our SASL/GSSAPI tests fail.

I'm going to specifically address failures in all-build-options via IMPALA-14530 with a more targeted fix to set IMPALA_MINIMAL_DEBUG_INFO=true, because that significantly reduces memory requirements and we don't need them in that case.

> Add support for limiting the concurrency of link jobs
> -----------------------------------------------------
>
>                 Key: IMPALA-13147
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13147
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Infrastructure
>    Affects Versions: Impala 4.5.0
>            Reporter: Joe McDonnell
>            Assignee: Michael Smith
>            Priority: Major
>
> Link jobs can use a lot of memory due to the amount of debug info. The level of concurrency that is useful for compilation can be too high for linking. Running a link-heavy command like buildall.sh -skiptests can run out of memory from linking all of the backend tests / benchmarks.
> It would be useful to be able to limit the number of concurrent link jobs. There are two basic approaches:
> When using the ninja generator for CMake, ninja supports having job pools with limited parallelism. CMake has support for mapping link tasks to their own pool. Here is an example:
> {noformat}
> set(CMAKE_JOB_POOLS compilation_pool=24 link_pool=8)
> set(CMAKE_JOB_POOL_COMPILE compilation_pool)
> set(CMAKE_JOB_POOL_LINK link_pool){noformat}
> The makefile generator does not have equivalent functionality, but we could do a more limited version where buildall.sh can split the -skiptests into two make invocations. The first does all the compilation with full parallelism (equivalent to -notests) and then the second make invocation does the backend tests / benchmarks with a reduced parallelism.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org