[jira] [Comment Edited] (MRESOLVER-32) Support parallel artifact/metadata uploads

["Tamás Cservenák (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MRESOLVER-32?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607557#comment-17607557 ] 

Tamás Cservenák edited comment on MRESOLVER-32 at 9/21/22 7:08 AM:
-------------------------------------------------------------------

Tricky at least.

The upload order within one module (so one module from a multi module build) is well defined with a reason: JAR/POM/metadata (the most common case), point is: metadata must be last. This is to circumvent some other build (consider a snapshot deploy and a CI or colleague starting another build) to pick partially performed deploy of artifact, as in case of snapshot, it relies on metadata (same for version ranges).

Hence, we could parallelize module uploads (while single module retains that order as described above), but alas, again: deploy happens as "things were built", so consider multi module build with modules M1 and M2 (where M2 depends on M1). Parallelization could in practice cause M2 to be deployed before M1, and again, it may lead to breakage on another concurrenclty running build, as M1 from same build is not yet deployed.

This may require some "smart deployer" in very same way as graph to build modules are established (like critical path based, see [http://takari.io/book/30-team-maven.html#takari-smart-builder)...]

Quality patches are welcome!

 


was (Author: cstamas):
Tricky at least.

The upload order within one module (so one module from a multi module build) is well defined with a reason: JAR/POM/metadata (the most common case), point is: metadata must be last. This is to circumvent some other build (consider a snapshot deploy and a CI or colleague starting another build) to pick partially performed deploys, as in case of snapshot, it relies on metadata (same for version ranges).

Hence, we could parallelize module uploads (while single module retains that order as described above), but alas, again: deploy happens as "things were built", so consider multi module build with modules M1 and M2 (where M2 depends on M1). Parallelization could in practice cause M2 to be deployed before M1, and again, it may lead to breakage on another concurrenclty running build, as M1 from same build is not yet deployed.

This may require some "smart deployer" in very same way as graph to build modules are established (like critical path based, see [http://takari.io/book/30-team-maven.html#takari-smart-builder)...]

Quality patches are welcome!

 

> Support parallel artifact/metadata uploads
> ------------------------------------------
>
>                 Key: MRESOLVER-32
>                 URL: https://issues.apache.org/jira/browse/MRESOLVER-32
>             Project: Maven Resolver
>          Issue Type: Improvement
>          Components: Resolver
>            Reporter: Matt Nelson
>            Priority: Major
>
> When using the {{maven-deploy-plugin}} with the {{deployAtEnd}} property on a large reactor the deployment process can take a considerable amount of time. I have a large reactor(50+ modules) where the upload takes 50% of the build time, 10 out of 20 minutes solely on the upload.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)