You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Tamas Cservenak (Jira)" <ji...@apache.org> on 2023/03/23 11:15:00 UTC
[jira] [Updated] (MRESOLVER-346) Too eager locking
[ https://issues.apache.org/jira/browse/MRESOLVER-346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Cservenak updated MRESOLVER-346:
--------------------------------------
Description:
The locking that is present in Resolver since 1.8.0 is too eager:
* there are no shared locks used at all
* this implies that local repository access is heavily serialized by locking
* there is no "upgrade" of locking due that above
This causes that "hot" artifacts in bigger multi module build run in parallel become bottleneck as all threads will wait for their moment to grab exclusive lock.
We can do it better: there are 4 main areas where locking is used:
* ArtifactInstaller: it is about to install (write) artifact files to lock repository, it needs exclusive lock, no change needed.
* ArtifactDeployer: it is about to upload present files to remote, it does not modifies anything on local. Change it's lock to shared. The exclusive lock also meant that if no DeployAtEnd used, other modules during resolution had to wait while this module uploaded.
* ArtifactResolver and MetadataResolver: two very similar beasts, they attempt to resolve locally (from local repo) w/o any content modification (read only), and if not successful, they will reach remote to download what is needed (write). Here we could do something similar to [DCL|https://en.wikipedia.org/wiki/Double-checked_locking] is: try with shared lock first, and if local content is not fulfilling, release shared, acquire exclusive and REDO all (as meanwhile someone else may downloaded files already).
was:
The locking that is present in Resolver since 1.8.0 is too eager:
* there are no shared locks used at all
* this implies that local repository access is heavily serialized by locking
* there is no "upgrade" of locking due that above
This causes that "hot" artifacts in bigger multi module build run in parallel become bottleneck as all threads will wait for their moment to grab exclusive lock.
We can do it better: there are 4 main areas where locking is used:
ArtifactInstaller: it is about to install (write) artifact files to lock repository, it needs exclusive lock, no change needed.
ArtifactDeployer: it is about to upload present files to remote, it does not modifies anything on local. Change it's lock to shared. The exclusive lock also meant that if no DeployAtEnd used, other modules during resolution had to wait while this module uploaded.
ArtifactResolver and MetadataResolver: two very similar beasts, they attempt to resolve locally (from local repo) w/o any content modification (read only), and if not successful, they will reach remote to download what is needed (write). Here we could do something similar to [DCL|https://en.wikipedia.org/wiki/Double-checked_locking] is: try with shared lock first, and if local content is not fulfilling, release shared, acquire exclusive and REDO all (as meanwhile someone else may downloaded files already).
> Too eager locking
> -----------------
>
> Key: MRESOLVER-346
> URL: https://issues.apache.org/jira/browse/MRESOLVER-346
> Project: Maven Resolver
> Issue Type: Bug
> Components: Resolver
> Reporter: Tamas Cservenak
> Priority: Major
> Fix For: 1.9.8
>
>
> The locking that is present in Resolver since 1.8.0 is too eager:
> * there are no shared locks used at all
> * this implies that local repository access is heavily serialized by locking
> * there is no "upgrade" of locking due that above
> This causes that "hot" artifacts in bigger multi module build run in parallel become bottleneck as all threads will wait for their moment to grab exclusive lock.
> We can do it better: there are 4 main areas where locking is used:
> * ArtifactInstaller: it is about to install (write) artifact files to lock repository, it needs exclusive lock, no change needed.
> * ArtifactDeployer: it is about to upload present files to remote, it does not modifies anything on local. Change it's lock to shared. The exclusive lock also meant that if no DeployAtEnd used, other modules during resolution had to wait while this module uploaded.
> * ArtifactResolver and MetadataResolver: two very similar beasts, they attempt to resolve locally (from local repo) w/o any content modification (read only), and if not successful, they will reach remote to download what is needed (write). Here we could do something similar to [DCL|https://en.wikipedia.org/wiki/Double-checked_locking] is: try with shared lock first, and if local content is not fulfilling, release shared, acquire exclusive and REDO all (as meanwhile someone else may downloaded files already).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)