You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "wei cai (Jira)" <ji...@apache.org> on 2021/12/06 03:04:00 UTC
[jira] [Updated] (MRESOLVER-229) Improve the download speed by downloading poms & jars in parallel as a warm-up
[ https://issues.apache.org/jira/browse/MRESOLVER-229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wei cai updated MRESOLVER-229:
------------------------------
Description:
We are tuning maven builds in our company, so far, we've made 2 big changes that we think they can benefit the opensource community.
* maven-resolver change by a skip & reconcile approach
https://issues.apache.org/jira/browse/MRESOLVER-228?filter=allopenissues
* download poms and jars in parallel ahead as a warm-up
It is not a new issue that downloading dependencies in maven is slow. Similar issues reported in:
https://issues.apache.org/jira/browse/MRESOLVER-7
Our solution is a bit different, basically we:
* Collect artifacts required for the project by listening to the maven dependency resolve event and store as a artifacts.json (GAV list & its repository ids) somewhere, in our case, we are storing the artifacts.json in a backend service.
* Before maven build kick starts, we called the maven-resolver api (DefaultDependencyCollector) to resolve all artifacts in artifacts.json in parallel.
* After warm-up completed, let maven still build the project as normal. This means even something missed in artifacts.json, maven could still find the missing part and resolve them.
This solution does not help for the very 1st build as no artifacts.json is built yet, however for any future builds, download can happen in parallel and thus it would be very quick.
This solution helps speed up the maven builds in below cases:
* All CI builds
* New hires or new machines as there is no local maven repo cache
* Snapshot updates
* When a new dependency added, especially for a dependency having lots of transitive dependencies
* Version upgrade such as Spring boot version upgrade
[~michael-o]
As a opensource solution, may be we can store the artifacts.json to .mvn in the git repository root, in this way, the artifacts list cache goes with the code.
Please share your comments with this solution.
was:
We are tuning maven builds in our company, so far, we've made 2 big changes that we think they can benefit the opensource community.
* maven-resolver change by a skip & reconcile approach
https://issues.apache.org/jira/browse/MRESOLVER-228?filter=allopenissues
* download poms and jars in parallel ahead as a warm-up
It is not a new issue that downloading dependencies in maven is slow. Similar issues reported in:
https://issues.apache.org/jira/browse/MRESOLVER-7
Our solution is a bit different, basically we:
* Collect artifacts required for the project by listening to the maven dependency resolve event and store as a artifacts.json (GAV list & its repository ids) somewhere, in our case, we are storing the artifacts.json in a backend service.
* Before maven build kick starts, we called the maven-resolver api (DefaultDependencyCollector) to resolve all artifacts in artifacts.json in parallel.
This solution does not help for the very 1st build as no artifacts.json is built yet, however for any future builds, download can happen in parallel and thus it would be very quick.
This solution helps speed up the maven builds in below cases:
* All CI builds
* New hires or new machines as there is no local maven repo cache
* Snapshot updates
* When a new dependency added, especially for a dependency having lots of transitive dependencies
* Version upgrade such as Spring boot version upgrade
[~michael-o]
As a opensource solution, may be we can store the artifacts.json to .mvn in the git repository root, in this way, the artifacts list cache goes with the code.
Please share your comments with this solution.
> Improve the download speed by downloading poms & jars in parallel as a warm-up
> ------------------------------------------------------------------------------
>
> Key: MRESOLVER-229
> URL: https://issues.apache.org/jira/browse/MRESOLVER-229
> Project: Maven Resolver
> Issue Type: Improvement
> Components: Resolver
> Affects Versions: 1.7.1
> Reporter: wei cai
> Priority: Major
>
> We are tuning maven builds in our company, so far, we've made 2 big changes that we think they can benefit the opensource community.
> * maven-resolver change by a skip & reconcile approach
> https://issues.apache.org/jira/browse/MRESOLVER-228?filter=allopenissues
> * download poms and jars in parallel ahead as a warm-up
> It is not a new issue that downloading dependencies in maven is slow. Similar issues reported in:
> https://issues.apache.org/jira/browse/MRESOLVER-7
> Our solution is a bit different, basically we:
> * Collect artifacts required for the project by listening to the maven dependency resolve event and store as a artifacts.json (GAV list & its repository ids) somewhere, in our case, we are storing the artifacts.json in a backend service.
> * Before maven build kick starts, we called the maven-resolver api (DefaultDependencyCollector) to resolve all artifacts in artifacts.json in parallel.
> * After warm-up completed, let maven still build the project as normal. This means even something missed in artifacts.json, maven could still find the missing part and resolve them.
> This solution does not help for the very 1st build as no artifacts.json is built yet, however for any future builds, download can happen in parallel and thus it would be very quick.
> This solution helps speed up the maven builds in below cases:
> * All CI builds
> * New hires or new machines as there is no local maven repo cache
> * Snapshot updates
> * When a new dependency added, especially for a dependency having lots of transitive dependencies
> * Version upgrade such as Spring boot version upgrade
> [~michael-o]
> As a opensource solution, may be we can store the artifacts.json to .mvn in the git repository root, in this way, the artifacts list cache goes with the code.
> Please share your comments with this solution.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)