[jira] (MNG-4706) Multithreaded building can create bad files for downloaded artifacts in local repository

["Tomas Kolda (JIRA)" <ji...@codehaus.org>]

    [ https://jira.codehaus.org/browse/MNG-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=346723#comment-346723 ] 

Tomas Kolda commented on MNG-4706:
----------------------------------

We were facing exactly same issue with 3.1.1. The problem is that Aether has empty default implementation of org.eclipse.aether.SyncContext that does not protect local repository at all. This is what happens:
1. Two modules depends on the same dependency
2. Resolution starts download process in more threads even if SyncContext.acquire was called for that artifact
3. The first thread creates the file using MOVE (temporary file->final file)
4. Second thread want to create the file MOVE (temporary file->final file), but it fails (move does not support overwrite). As a fallback it simply do copy over.
5. First thread fails, because it gets invalid data (half of jar file, etc.)

This must not happen if Maven is declaring parallel build support. There are some projects like https://github.com/grdryn/tesla-concurrent-localrepo that introduce file locking using lockfile.I am not sure if it has some impact on performance due to I/O (create/delete files).

I wrote very simple (one class) memory only locking that works too and does not depend on anything else. It does not work for multiple processes, but it works in similar way.

Maven should have at least some basic (one process only) implementation of this SyncContext so that users will not need to update Maven instalation using lib/ext.

> Multithreaded building can create bad files for downloaded artifacts in local repository
> ----------------------------------------------------------------------------------------
>
>                 Key: MNG-4706
>                 URL: https://jira.codehaus.org/browse/MNG-4706
>             Project: Maven 2 & 3
>          Issue Type: Bug
>          Components: Artifacts and Repositories
>    Affects Versions: 3.0-beta-1
>            Reporter: Kristian Rosenvold
>            Assignee: Kristian Rosenvold
>             Fix For: Issues to be reviewed for 4.x
>
>
> Within a parallel build, modules A and B can start at the same time and decide at the same time that they need to download artifact X. Since X is not available locally, both can start download of X. Both will download X and both will try to write the file for X to disk.
> On Windows platforms, one of them can fail with file locking problems. On Unixes the contents of the written file can be non-deterministic.
> This issue is related to MNG-2802 in the sense that it can create a corrupted artifact file in the local repository, and it can happen to ANY file written. But as opposed to MNG-2802 the synhronization needed to fix this issue is basically only within a single running maven process, so regular java mechanisms can apply, whereas MNG-2802 probably needs to use a totally different solution space.



--
This message was sent by Atlassian JIRA
(v6.1.6#6162)