[jira] [Updated] (BIGTOP-4044) Enhance Bigtop with Concurrent Compilation Support for Additional Components

["Masatake Iwasaki (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BIGTOP-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Masatake Iwasaki updated BIGTOP-4044:
-------------------------------------
    Fix Version/s: 3.3.0

> Enhance Bigtop with Concurrent Compilation Support for Additional Components
> ----------------------------------------------------------------------------
>
>                 Key: BIGTOP-4044
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-4044
>             Project: Bigtop
>          Issue Type: Improvement
>            Reporter: caijialiang
>            Assignee: caijialiang
>            Priority: Major
>             Fix For: 3.3.0
>
>          Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
>  
> Background: In the components maintained by Bigtop, the majority are Java-based technologies, and they are built using Maven.
> Necessity: Compiling components with a large number of modules can be slow. For individual components with hundreds of modules, compiling them one by one is time-consuming. Even when all dependencies are pre-downloaded during the second compilation, it remains slow due to sequential compilation. Even if all components are compiled together, they are compiled sequentially, which makes it challenging to fully utilize CPU resources and does not significantly reduce compilation time. As a result, repetitive compilation and testing lead to significant wait times.
> Therefore, I propose adding a new parameter that allows users to enable or disable parallel compilation as needed for components compiled using Maven.
> The related pull request (PR) can be roughly divided into two parts:
>  # The first part involves adding parallel compilation functionality and enabling parallel compilation for components that have been tested and do not experience any additional issues with parallel compilation. These components may include Hive, HBase, Flink, ZooKeeper, and others.
>  # The second part involves enabling parallel compilation for components that encounter issues with parallel compilation and require additional patches to fix Maven's parallel compilation capabilities. These components may include Ranger, Tez, Hadoop, Spark, and others.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)