You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Francois Marot <fr...@gmail.com> on 2021/10/04 12:31:04 UTC
Local repository accessed by multiple Maven instances - how is it
solved ?
Hello all,
I would like clarifications on MNG-2802[*] that seems to be solved. If I
understand correctly it solves the problem where multiple simultaneous
Maven executions shared the same local repository.
I have been keeping for years a bit of code in my Jenkinsfiles ensuring the
local repos were accessed only by a single jenkins executor at the same
time, so it seems like I can get rid of this doesn't it ?
I'd like your input on this because reproducing the bug is difficult so I'd
like to be sure before simplifying my Jenkinsfile.
Moreover, does anyone know how this problem is solved technically ? Using
files lock ? Can anybody explain ?
Thanks you and thanks the Maven team for keeping up the good work at such
pace !
François MAROT
* https://issues.apache.org/jira/browse/MNG-2802
Re: Local repository accessed by multiple Maven instances - how is it solved ?
Posted by Michael Osipov <mi...@apache.org>.
* Build this one first:
https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x
(https://maven.apache.org/resolver/maven-3.8.x.html)
* Follow this guide:
https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
If you have failures, follow this guide:
*
https://maven.apache.org/resolver/maven-resolver-named-locks/analyzing-lock-issues.html
The guide has been tested on realworld projects reported by users.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it solved ?
Posted by Delany <de...@gmail.com>.
Hi Tamas,
I've been trying the Redisson approach for the last 7 weeks, but my machine
locks up every second or third build for long periods of time 5-20min.
I'm on Ubuntu running the Redis 6.2.6 snap using the Maven 3.8.3 and 3.8.4
wrapper. I followed these instructions to setup
Edit the file *${maven.home}/bin/m2.conf* and after *${maven.conf}/logging*
add *load ${maven.home}/lib/ext/redisson/*.jar*
Create the folder *${maven.home}/lib/ext/redisson/* and populate with
https://repo1.maven.org/maven2/org/apache/maven/resolver/maven-resolver-named-locks-redisson/1.7.2/maven-resolver-named-locks-redisson-1.7.2.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.1/jackson-annotations-2.12.1.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.1/jackson-core-2.12.1.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.1/jackson-databind-2.12.1.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.12.1/jackson-dataformat-yaml-2.12.1.jar
https://repo1.maven.org/maven2/org/jboss/marshalling/jboss-marshalling/2.0.11.Final/jboss-marshalling-2.0.11.Final.jar
https://repo1.maven.org/maven2/org/jboss/marshalling/jboss-marshalling-river/2.0.11.Final/jboss-marshalling-river-2.0.11.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.65.Final/netty-buffer-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.65.Final/netty-codec-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-codec-dns/4.1.65.Final/netty-codec-dns-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-common/4.1.65.Final/netty-common-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.65.Final/netty-handler-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.65.Final/netty-resolver-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-resolver-dns/4.1.65.Final/netty-resolver-dns-4.1.65.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.65.Final/netty-transport-4.1.65.Final.jar
https://repo1.maven.org/maven2/org/redisson/redisson/3.15.6/redisson-3.15.6.jar
https://repo1.maven.org/maven2/org/yaml/snakeyaml/1.27/snakeyaml-1.27.jar
Delany
On Fri, 8 Oct 2021 at 10:05, Delany <de...@gmail.com> wrote:
> Thanks Tamás.
> Since I use the wrapper, I was thinking there's a case for Maven wrapper
> variants. But I guess this is what containers are all about anyway.
> Delany
>
> On Thu, 7 Oct 2021 at 11:32, Tamás Cservenák <ta...@cservenak.net> wrote:
>
>> Hi Delany,
>>
>> from Sisu website: "Sisu is a modular JSR330-based container that
>> supports *classpath
>> scanning*, *auto-binding*, and *dynamic auto-wiring*. Sisu uses
>> Google-Guice to perform dependency injection and provide the core JSR330
>> support, but removes the need to write explicit bindings in Guice
>> modules.".
>>
>> Simply, SISU will scan your classpath for components and dynamically pick
>> them up and wire them up.
>>
>> For Guice, you need to _explicitly add_ bindings (as this "dynamism" is a
>> Sisu feature).
>>
>> ServiceLocator is deprecated, leave it out (but same thing stands: is
>> "static").
>>
>> ===
>>
>> More simpler: with Sisu, just throw a component onto classpath, and Sisu
>> will "automagically" pick it up, no code change needed. This is how
>> extensions and other things work in Maven for example (just copy a JAR
>> with
>> components to classpath and they are discovered).
>>
>> Locking implementations like Redisson and Hazelcast are "opt-in", they are
>> not bundled with Maven (to not bloat distro). They work only after you
>> "install" them (provide JARs with their components and all component
>> dependencies to Maven classpath, as explained in the Install step on that
>> page).
>>
>> This remark you quoted is geared more toward "resolver integrators", for
>> those using Resolver into some other project than Maven. If you integrate
>> Resolver, then:
>> - if you use Sisu, you have nothing to do :)
>> - if you use vanilla Guice, then in your integration you have to provide
>> the things you want on classpath and explicitly bind them
>> - if you use ServiceLocator, move away from it (will be soon dropped),
>> simplest is to "roll your own" bootstrap class (that does same as
>> ServiceLocator was doing, statically wiring things up), and again, you
>> should upfront provide whatever you need on classpath and instantiate them
>> using your own "bootstrap" class.
>>
>> If you just want to use it with Maven, nothing to be done for you, just
>> Install it as per page.
>>
>> ===
>>
>> HTH
>> Tamas
>>
>> On Thu, Oct 7, 2021 at 10:07 AM Delany <de...@gmail.com>
>> wrote:
>>
>> > Michael, could you clarify this line pls: "It only works when Sisu DI is
>> > used and not the bundled AetherModule or ServiceLocator (Maven uses Sisu
>> > dependency injection)."
>> >
>> >
>> https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
>> >
>> > What should I do about this issue?
>> >
>> > Thanks,
>> > Delany
>> >
>> >
>> > On Wed, 6 Oct 2021 at 22:18, Michael Osipov <mi...@apache.org>
>> wrote:
>> >
>> > > Am 2021-10-06 um 21:05 schrieb Francois Marot:
>> > > > Michael, I do not agree. As a user and maintainer of the build
>> chain in
>> > > my
>> > > > company, I think Maven would really benefit from an out of the box
>> > > > solution. I think any user is susceptible to the bug by launching
>> > > multiple
>> > > > Maven instances at the same time on his computer. Bugs then
>> encountered
>> > > are
>> > > > really a bad sign sent to users, and requiring each dev to setup a
>> > > database
>> > > > (even simple) is cumbersome.
>> > >
>> > > I agree with you, but we are developers after all, I can expect people
>> > > to set up something if necessary.
>> > > At the end you need to consider that file based locking has issues, it
>> > > does not work everywhere, it is advisory at best, it protects you only
>> > > from multiprocess access, multithreaded requires an extra layer of
>> > > in-JVM locks. Given that we are all volunteers and the pain for the
>> > > community isn't big enough to contribute something valuable, I won't.
>> > > Especially that Tamás and me invested almost year developing the named
>> > > locks approach with pluggable providers is more than enough to solve
>> any
>> > > type of build setup on any FS and OS.
>> > >
>> > > I'd like to quote Marshall Kirk McKusick: Why write something good if
>> > > you can steal something better? (trade file locks with Redis).
>> > >
>> > > A more lightweight approach would be something like POSIX semaphores
>> > > available everywhere, but Windows. Requires likely native code or JNR
>> > > and time. Out of personal interest I'd peek at it next year.
>> > >
>> > > M
>> > >
>> > > > Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a
>> > > écrit :
>> > > >
>> > > >> Tamás,
>> > > >>
>> > > >> Redis is so easy to install and get going in 5 minutes that I would
>> > > >> rather see your energy go into areas which need more attention.
>> > > >>
>> > > >> M
>> > > >>
>> > > >> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
>> > > >>> Hi Bernd,
>> > > >>>
>> > > >>> nothing is wrong with advisory file locking, as long as you don't
>> > store
>> > > >>> local repo on NFS ;)
>> > > >>> Will re-add file locking once I get there, as in my opinion it is
>> the
>> > > >> most
>> > > >>> "lightweight" MP (multi process) solution on a single host.
>> > > >>> Redis and Hazelcast are more for "farms", where several hosts with
>> > many
>> > > >>> processes (and each with many threads) is bashing local repo (that
>> > MAY
>> > > be
>> > > >>> on NFS as well).
>> > > >>>
>> > > >>> Thanks
>> > > >>> Tamas
>> > > >>>
>> > > >>> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <
>> > ecki@zusammenkunft.net
>> > > >
>> > > >>> wrote:
>> > > >>>
>> > > >>>> What’s the problem with adivisory locking, as long as Maven
>> honors
>> > the
>> > > >>>> advice it is the same as it’s a redis lock? (But much less
>> > footprint).
>> > > >> In
>> > > >>>> fact on the same machine it should even work without locking as
>> Long
>> > > as
>> > > >> you
>> > > >>>> use pidfiles?
>> > > >>>>
>> > > >>>> Gruss
>> > > >>>> Bernd
>> > > >>>>
>> > > >>>>
>> > > >>>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> > > >> For additional commands, e-mail: users-help@maven.apache.org
>> > > >>
>> > > >>
>> > > >
>> > >
>> > >
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> > > For additional commands, e-mail: users-help@maven.apache.org
>> > >
>> > >
>> >
>>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Delany <de...@gmail.com>.
Thanks Tamás.
Since I use the wrapper, I was thinking there's a case for Maven wrapper
variants. But I guess this is what containers are all about anyway.
Delany
On Thu, 7 Oct 2021 at 11:32, Tamás Cservenák <ta...@cservenak.net> wrote:
> Hi Delany,
>
> from Sisu website: "Sisu is a modular JSR330-based container that
> supports *classpath
> scanning*, *auto-binding*, and *dynamic auto-wiring*. Sisu uses
> Google-Guice to perform dependency injection and provide the core JSR330
> support, but removes the need to write explicit bindings in Guice
> modules.".
>
> Simply, SISU will scan your classpath for components and dynamically pick
> them up and wire them up.
>
> For Guice, you need to _explicitly add_ bindings (as this "dynamism" is a
> Sisu feature).
>
> ServiceLocator is deprecated, leave it out (but same thing stands: is
> "static").
>
> ===
>
> More simpler: with Sisu, just throw a component onto classpath, and Sisu
> will "automagically" pick it up, no code change needed. This is how
> extensions and other things work in Maven for example (just copy a JAR with
> components to classpath and they are discovered).
>
> Locking implementations like Redisson and Hazelcast are "opt-in", they are
> not bundled with Maven (to not bloat distro). They work only after you
> "install" them (provide JARs with their components and all component
> dependencies to Maven classpath, as explained in the Install step on that
> page).
>
> This remark you quoted is geared more toward "resolver integrators", for
> those using Resolver into some other project than Maven. If you integrate
> Resolver, then:
> - if you use Sisu, you have nothing to do :)
> - if you use vanilla Guice, then in your integration you have to provide
> the things you want on classpath and explicitly bind them
> - if you use ServiceLocator, move away from it (will be soon dropped),
> simplest is to "roll your own" bootstrap class (that does same as
> ServiceLocator was doing, statically wiring things up), and again, you
> should upfront provide whatever you need on classpath and instantiate them
> using your own "bootstrap" class.
>
> If you just want to use it with Maven, nothing to be done for you, just
> Install it as per page.
>
> ===
>
> HTH
> Tamas
>
> On Thu, Oct 7, 2021 at 10:07 AM Delany <de...@gmail.com> wrote:
>
> > Michael, could you clarify this line pls: "It only works when Sisu DI is
> > used and not the bundled AetherModule or ServiceLocator (Maven uses Sisu
> > dependency injection)."
> >
> >
> https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
> >
> > What should I do about this issue?
> >
> > Thanks,
> > Delany
> >
> >
> > On Wed, 6 Oct 2021 at 22:18, Michael Osipov <mi...@apache.org> wrote:
> >
> > > Am 2021-10-06 um 21:05 schrieb Francois Marot:
> > > > Michael, I do not agree. As a user and maintainer of the build chain
> in
> > > my
> > > > company, I think Maven would really benefit from an out of the box
> > > > solution. I think any user is susceptible to the bug by launching
> > > multiple
> > > > Maven instances at the same time on his computer. Bugs then
> encountered
> > > are
> > > > really a bad sign sent to users, and requiring each dev to setup a
> > > database
> > > > (even simple) is cumbersome.
> > >
> > > I agree with you, but we are developers after all, I can expect people
> > > to set up something if necessary.
> > > At the end you need to consider that file based locking has issues, it
> > > does not work everywhere, it is advisory at best, it protects you only
> > > from multiprocess access, multithreaded requires an extra layer of
> > > in-JVM locks. Given that we are all volunteers and the pain for the
> > > community isn't big enough to contribute something valuable, I won't.
> > > Especially that Tamás and me invested almost year developing the named
> > > locks approach with pluggable providers is more than enough to solve
> any
> > > type of build setup on any FS and OS.
> > >
> > > I'd like to quote Marshall Kirk McKusick: Why write something good if
> > > you can steal something better? (trade file locks with Redis).
> > >
> > > A more lightweight approach would be something like POSIX semaphores
> > > available everywhere, but Windows. Requires likely native code or JNR
> > > and time. Out of personal interest I'd peek at it next year.
> > >
> > > M
> > >
> > > > Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a
> > > écrit :
> > > >
> > > >> Tamás,
> > > >>
> > > >> Redis is so easy to install and get going in 5 minutes that I would
> > > >> rather see your energy go into areas which need more attention.
> > > >>
> > > >> M
> > > >>
> > > >> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
> > > >>> Hi Bernd,
> > > >>>
> > > >>> nothing is wrong with advisory file locking, as long as you don't
> > store
> > > >>> local repo on NFS ;)
> > > >>> Will re-add file locking once I get there, as in my opinion it is
> the
> > > >> most
> > > >>> "lightweight" MP (multi process) solution on a single host.
> > > >>> Redis and Hazelcast are more for "farms", where several hosts with
> > many
> > > >>> processes (and each with many threads) is bashing local repo (that
> > MAY
> > > be
> > > >>> on NFS as well).
> > > >>>
> > > >>> Thanks
> > > >>> Tamas
> > > >>>
> > > >>> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <
> > ecki@zusammenkunft.net
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> What’s the problem with adivisory locking, as long as Maven honors
> > the
> > > >>>> advice it is the same as it’s a redis lock? (But much less
> > footprint).
> > > >> In
> > > >>>> fact on the same machine it should even work without locking as
> Long
> > > as
> > > >> you
> > > >>>> use pidfiles?
> > > >>>>
> > > >>>> Gruss
> > > >>>> Bernd
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >>
> ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > > >> For additional commands, e-mail: users-help@maven.apache.org
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: users-help@maven.apache.org
> > >
> > >
> >
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Tamás Cservenák <ta...@cservenak.net>.
Hi Delany,
from Sisu website: "Sisu is a modular JSR330-based container that
supports *classpath
scanning*, *auto-binding*, and *dynamic auto-wiring*. Sisu uses
Google-Guice to perform dependency injection and provide the core JSR330
support, but removes the need to write explicit bindings in Guice modules.".
Simply, SISU will scan your classpath for components and dynamically pick
them up and wire them up.
For Guice, you need to _explicitly add_ bindings (as this "dynamism" is a
Sisu feature).
ServiceLocator is deprecated, leave it out (but same thing stands: is
"static").
===
More simpler: with Sisu, just throw a component onto classpath, and Sisu
will "automagically" pick it up, no code change needed. This is how
extensions and other things work in Maven for example (just copy a JAR with
components to classpath and they are discovered).
Locking implementations like Redisson and Hazelcast are "opt-in", they are
not bundled with Maven (to not bloat distro). They work only after you
"install" them (provide JARs with their components and all component
dependencies to Maven classpath, as explained in the Install step on that
page).
This remark you quoted is geared more toward "resolver integrators", for
those using Resolver into some other project than Maven. If you integrate
Resolver, then:
- if you use Sisu, you have nothing to do :)
- if you use vanilla Guice, then in your integration you have to provide
the things you want on classpath and explicitly bind them
- if you use ServiceLocator, move away from it (will be soon dropped),
simplest is to "roll your own" bootstrap class (that does same as
ServiceLocator was doing, statically wiring things up), and again, you
should upfront provide whatever you need on classpath and instantiate them
using your own "bootstrap" class.
If you just want to use it with Maven, nothing to be done for you, just
Install it as per page.
===
HTH
Tamas
On Thu, Oct 7, 2021 at 10:07 AM Delany <de...@gmail.com> wrote:
> Michael, could you clarify this line pls: "It only works when Sisu DI is
> used and not the bundled AetherModule or ServiceLocator (Maven uses Sisu
> dependency injection)."
>
> https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
>
> What should I do about this issue?
>
> Thanks,
> Delany
>
>
> On Wed, 6 Oct 2021 at 22:18, Michael Osipov <mi...@apache.org> wrote:
>
> > Am 2021-10-06 um 21:05 schrieb Francois Marot:
> > > Michael, I do not agree. As a user and maintainer of the build chain in
> > my
> > > company, I think Maven would really benefit from an out of the box
> > > solution. I think any user is susceptible to the bug by launching
> > multiple
> > > Maven instances at the same time on his computer. Bugs then encountered
> > are
> > > really a bad sign sent to users, and requiring each dev to setup a
> > database
> > > (even simple) is cumbersome.
> >
> > I agree with you, but we are developers after all, I can expect people
> > to set up something if necessary.
> > At the end you need to consider that file based locking has issues, it
> > does not work everywhere, it is advisory at best, it protects you only
> > from multiprocess access, multithreaded requires an extra layer of
> > in-JVM locks. Given that we are all volunteers and the pain for the
> > community isn't big enough to contribute something valuable, I won't.
> > Especially that Tamás and me invested almost year developing the named
> > locks approach with pluggable providers is more than enough to solve any
> > type of build setup on any FS and OS.
> >
> > I'd like to quote Marshall Kirk McKusick: Why write something good if
> > you can steal something better? (trade file locks with Redis).
> >
> > A more lightweight approach would be something like POSIX semaphores
> > available everywhere, but Windows. Requires likely native code or JNR
> > and time. Out of personal interest I'd peek at it next year.
> >
> > M
> >
> > > Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a
> > écrit :
> > >
> > >> Tamás,
> > >>
> > >> Redis is so easy to install and get going in 5 minutes that I would
> > >> rather see your energy go into areas which need more attention.
> > >>
> > >> M
> > >>
> > >> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
> > >>> Hi Bernd,
> > >>>
> > >>> nothing is wrong with advisory file locking, as long as you don't
> store
> > >>> local repo on NFS ;)
> > >>> Will re-add file locking once I get there, as in my opinion it is the
> > >> most
> > >>> "lightweight" MP (multi process) solution on a single host.
> > >>> Redis and Hazelcast are more for "farms", where several hosts with
> many
> > >>> processes (and each with many threads) is bashing local repo (that
> MAY
> > be
> > >>> on NFS as well).
> > >>>
> > >>> Thanks
> > >>> Tamas
> > >>>
> > >>> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <
> ecki@zusammenkunft.net
> > >
> > >>> wrote:
> > >>>
> > >>>> What’s the problem with adivisory locking, as long as Maven honors
> the
> > >>>> advice it is the same as it’s a redis lock? (But much less
> footprint).
> > >> In
> > >>>> fact on the same machine it should even work without locking as Long
> > as
> > >> you
> > >>>> use pidfiles?
> > >>>>
> > >>>> Gruss
> > >>>> Bernd
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > >> For additional commands, e-mail: users-help@maven.apache.org
> > >>
> > >>
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Delany <de...@gmail.com>.
Michael, could you clarify this line pls: "It only works when Sisu DI is
used and not the bundled AetherModule or ServiceLocator (Maven uses Sisu
dependency injection)."
https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
What should I do about this issue?
Thanks,
Delany
On Wed, 6 Oct 2021 at 22:18, Michael Osipov <mi...@apache.org> wrote:
> Am 2021-10-06 um 21:05 schrieb Francois Marot:
> > Michael, I do not agree. As a user and maintainer of the build chain in
> my
> > company, I think Maven would really benefit from an out of the box
> > solution. I think any user is susceptible to the bug by launching
> multiple
> > Maven instances at the same time on his computer. Bugs then encountered
> are
> > really a bad sign sent to users, and requiring each dev to setup a
> database
> > (even simple) is cumbersome.
>
> I agree with you, but we are developers after all, I can expect people
> to set up something if necessary.
> At the end you need to consider that file based locking has issues, it
> does not work everywhere, it is advisory at best, it protects you only
> from multiprocess access, multithreaded requires an extra layer of
> in-JVM locks. Given that we are all volunteers and the pain for the
> community isn't big enough to contribute something valuable, I won't.
> Especially that Tamás and me invested almost year developing the named
> locks approach with pluggable providers is more than enough to solve any
> type of build setup on any FS and OS.
>
> I'd like to quote Marshall Kirk McKusick: Why write something good if
> you can steal something better? (trade file locks with Redis).
>
> A more lightweight approach would be something like POSIX semaphores
> available everywhere, but Windows. Requires likely native code or JNR
> and time. Out of personal interest I'd peek at it next year.
>
> M
>
> > Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a
> écrit :
> >
> >> Tamás,
> >>
> >> Redis is so easy to install and get going in 5 minutes that I would
> >> rather see your energy go into areas which need more attention.
> >>
> >> M
> >>
> >> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
> >>> Hi Bernd,
> >>>
> >>> nothing is wrong with advisory file locking, as long as you don't store
> >>> local repo on NFS ;)
> >>> Will re-add file locking once I get there, as in my opinion it is the
> >> most
> >>> "lightweight" MP (multi process) solution on a single host.
> >>> Redis and Hazelcast are more for "farms", where several hosts with many
> >>> processes (and each with many threads) is bashing local repo (that MAY
> be
> >>> on NFS as well).
> >>>
> >>> Thanks
> >>> Tamas
> >>>
> >>> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ecki@zusammenkunft.net
> >
> >>> wrote:
> >>>
> >>>> What’s the problem with adivisory locking, as long as Maven honors the
> >>>> advice it is the same as it’s a redis lock? (But much less footprint).
> >> In
> >>>> fact on the same machine it should even work without locking as Long
> as
> >> you
> >>>> use pidfiles?
> >>>>
> >>>> Gruss
> >>>> Bernd
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: users-help@maven.apache.org
> >>
> >>
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Michael Osipov <mi...@apache.org>.
Am 2021-10-06 um 21:05 schrieb Francois Marot:
> Michael, I do not agree. As a user and maintainer of the build chain in my
> company, I think Maven would really benefit from an out of the box
> solution. I think any user is susceptible to the bug by launching multiple
> Maven instances at the same time on his computer. Bugs then encountered are
> really a bad sign sent to users, and requiring each dev to setup a database
> (even simple) is cumbersome.
I agree with you, but we are developers after all, I can expect people
to set up something if necessary.
At the end you need to consider that file based locking has issues, it
does not work everywhere, it is advisory at best, it protects you only
from multiprocess access, multithreaded requires an extra layer of
in-JVM locks. Given that we are all volunteers and the pain for the
community isn't big enough to contribute something valuable, I won't.
Especially that Tamás and me invested almost year developing the named
locks approach with pluggable providers is more than enough to solve any
type of build setup on any FS and OS.
I'd like to quote Marshall Kirk McKusick: Why write something good if
you can steal something better? (trade file locks with Redis).
A more lightweight approach would be something like POSIX semaphores
available everywhere, but Windows. Requires likely native code or JNR
and time. Out of personal interest I'd peek at it next year.
M
> Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a écrit :
>
>> Tamás,
>>
>> Redis is so easy to install and get going in 5 minutes that I would
>> rather see your energy go into areas which need more attention.
>>
>> M
>>
>> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
>>> Hi Bernd,
>>>
>>> nothing is wrong with advisory file locking, as long as you don't store
>>> local repo on NFS ;)
>>> Will re-add file locking once I get there, as in my opinion it is the
>> most
>>> "lightweight" MP (multi process) solution on a single host.
>>> Redis and Hazelcast are more for "farms", where several hosts with many
>>> processes (and each with many threads) is bashing local repo (that MAY be
>>> on NFS as well).
>>>
>>> Thanks
>>> Tamas
>>>
>>> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ec...@zusammenkunft.net>
>>> wrote:
>>>
>>>> What’s the problem with adivisory locking, as long as Maven honors the
>>>> advice it is the same as it’s a redis lock? (But much less footprint).
>> In
>>>> fact on the same machine it should even work without locking as Long as
>> you
>>>> use pidfiles?
>>>>
>>>> Gruss
>>>> Bernd
>>>>
>>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
>> For additional commands, e-mail: users-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Francois Marot <fr...@gmail.com>.
Michael, I do not agree. As a user and maintainer of the build chain in my
company, I think Maven would really benefit from an out of the box
solution. I think any user is susceptible to the bug by launching multiple
Maven instances at the same time on his computer. Bugs then encountered are
really a bad sign sent to users, and requiring each dev to setup a database
(even simple) is cumbersome.
Regards
François
Le lun. 4 oct. 2021 à 22:13, Michael Osipov <mi...@apache.org> a écrit :
> Tamás,
>
> Redis is so easy to install and get going in 5 minutes that I would
> rather see your energy go into areas which need more attention.
>
> M
>
> Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
> > Hi Bernd,
> >
> > nothing is wrong with advisory file locking, as long as you don't store
> > local repo on NFS ;)
> > Will re-add file locking once I get there, as in my opinion it is the
> most
> > "lightweight" MP (multi process) solution on a single host.
> > Redis and Hazelcast are more for "farms", where several hosts with many
> > processes (and each with many threads) is bashing local repo (that MAY be
> > on NFS as well).
> >
> > Thanks
> > Tamas
> >
> > On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ec...@zusammenkunft.net>
> > wrote:
> >
> >> What’s the problem with adivisory locking, as long as Maven honors the
> >> advice it is the same as it’s a redis lock? (But much less footprint).
> In
> >> fact on the same machine it should even work without locking as Long as
> you
> >> use pidfiles?
> >>
> >> Gruss
> >> Bernd
> >>
> >>
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Michael Osipov <mi...@apache.org>.
Tamás,
Redis is so easy to install and get going in 5 minutes that I would
rather see your energy go into areas which need more attention.
M
Am 2021-10-04 um 22:02 schrieb Tamás Cservenák:
> Hi Bernd,
>
> nothing is wrong with advisory file locking, as long as you don't store
> local repo on NFS ;)
> Will re-add file locking once I get there, as in my opinion it is the most
> "lightweight" MP (multi process) solution on a single host.
> Redis and Hazelcast are more for "farms", where several hosts with many
> processes (and each with many threads) is bashing local repo (that MAY be
> on NFS as well).
>
> Thanks
> Tamas
>
> On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ec...@zusammenkunft.net>
> wrote:
>
>> What’s the problem with adivisory locking, as long as Maven honors the
>> advice it is the same as it’s a redis lock? (But much less footprint). In
>> fact on the same machine it should even work without locking as Long as you
>> use pidfiles?
>>
>> Gruss
>> Bernd
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Even for nfs fcntl locking can work and create/ln/mv are atomic. Do you know if there is any file lock based impl available?
Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
Von: Tamás Cservenák <ta...@cservenak.net>
Gesendet: Monday, October 4, 2021 10:02:27 PM
An: Maven Users List <us...@maven.apache.org>
Betreff: Re: Local repository accessed by multiple Maven instances - how is it solved ?
Hi Bernd,
nothing is wrong with advisory file locking, as long as you don't store
local repo on NFS ;)
Will re-add file locking once I get there, as in my opinion it is the most
"lightweight" MP (multi process) solution on a single host.
Redis and Hazelcast are more for "farms", where several hosts with many
processes (and each with many threads) is bashing local repo (that MAY be
on NFS as well).
Thanks
Tamas
On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ec...@zusammenkunft.net>
wrote:
> What’s the problem with adivisory locking, as long as Maven honors the
> advice it is the same as it’s a redis lock? (But much less footprint). In
> fact on the same machine it should even work without locking as Long as you
> use pidfiles?
>
> Gruss
> Bernd
>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Tamás Cservenák <ta...@cservenak.net>.
Hi Bernd,
nothing is wrong with advisory file locking, as long as you don't store
local repo on NFS ;)
Will re-add file locking once I get there, as in my opinion it is the most
"lightweight" MP (multi process) solution on a single host.
Redis and Hazelcast are more for "farms", where several hosts with many
processes (and each with many threads) is bashing local repo (that MAY be
on NFS as well).
Thanks
Tamas
On Mon, Oct 4, 2021 at 9:37 PM Bernd Eckenfels <ec...@zusammenkunft.net>
wrote:
> What’s the problem with adivisory locking, as long as Maven honors the
> advice it is the same as it’s a redis lock? (But much less footprint). In
> fact on the same machine it should even work without locking as Long as you
> use pidfiles?
>
> Gruss
> Bernd
>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
What’s the problem with adivisory locking, as long as Maven honors the advice it is the same as it’s a redis lock? (But much less footprint). In fact on the same machine it should even work without locking as Long as you use pidfiles?
Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
Von: Michael Osipov <mi...@apache.org>
Gesendet: Monday, October 4, 2021 9:15:23 PM
An: users@maven.apache.org <us...@maven.apache.org>
Betreff: Re: Local repository accessed by multiple Maven instances - how is it solved ?
Am 2021-10-04 um 14:31 schrieb Francois Marot:
> Hello all,
>
> I would like clarifications on MNG-2802[*] that seems to be solved. If I
> understand correctly it solves the problem where multiple simultaneous
> Maven executions shared the same local repository.
> I have been keeping for years a bit of code in my Jenkinsfiles ensuring the
> local repos were accessed only by a single jenkins executor at the same
> time, so it seems like I can get rid of this doesn't it ?
> I'd like your input on this because reproducing the bug is difficult so I'd
> like to be sure before simplifying my Jenkinsfile.
>
> Moreover, does anyone know how this problem is solved technically ? Using
> files lock ? Can anybody explain ?
>
> Thanks you and thanks the Maven team for keeping up the good work at such
> pace !
Salut François,
I don't know what you exactly fiddled in your Jenkinsfile, but there is
no way make it right just with Jenkinsfile foo when you want access
*one* local repo with *multiple* processes you *must* use an external
coordinator process.
Forget also about file locks, they are all advisory, at least on
POSIX-like filesystems. There is no way, or a very hard way to make them
right. We tried and removed them.
Now, let me give you bit of history: I have integrated/implemented a
fully working multi-process Redisson-based synchronization lock factory
for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already
been battle tested for several months by a user on a busy CI server).
Tamás took it to the next level in Resolver 1.7.x and generalized it to
a named lock approach where the Redisson Lock Factory plugs in nicely
with other lock implementations. Almost nine months of work and it works
very good now.
Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x.
I have created a branch which reconciles best of breed: Maven 3.8.x and
Maven Resolver 1.7.x. It just works (also verified by this user). I also
ran more than a thousand builds with 8 parallel processes and 4 threads.
It will easily handle thousands of locks in parralel with an one digit
percent overhead.
I don't know for what documentation you are waiting for, but everything
is there, you can start *now* to leverage it: [2], [3]. With Resolver
2.0.0 and Maven 4.0.0 it will likely be even easier to integrate.
Hazelcast will also work, but Tamás is the special here.
Again: You *must* use an external process to synchronize access. No excuses.
Michael
[1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x
[2] https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
[3]
https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Michael Osipov <mi...@apache.org>.
Am 2021-10-04 um 22:13 schrieb Francois Marot:
> "Salut" Michael,
>
> thanks for the detailled answer.
> Regarding my Jenkins 'hack' you can totally forget about it: it just
> ensures no more than one Maven execution uses a given repo at once by
> locating the local repositories in a folder reflecting the executor (which
> are kinda Jenkins "threads").
> I plan not to use your work right now mostly because it would involve
> setting up a Redis Database and my 'hack' still works (despite using a lot
> of disk space and being a bit slow).
>
> If I understand correctly after Tams' message, your work is mostly geared
> toward running multiple Maven build in parallel and on multiple hosts, all
> sharing one big "local" repository through the network. Am I correct ? In
> my case, I have multiple Jenkins workers but never thought about making
> them collaborate on a shared "local" repo. I'll have to think about it !
Not quite, we have multiple named locks which will give you the
granularity you need. Either in-JVM only, or multi-JVM. Then it does not
matter whether it is on the same host or not. You decide, the Redisson
approach will work. You need to read the docs how named locks are
discrimnated with a lock impl. That is all. You will benefit. User I was
talking about has now a huge speed bump due to the shared local repo,
for free.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it solved ?
Posted by Jacques Etienne Beaudet <je...@gmail.com>.
Just chiming in on my experience using the redis locks with maven 3.8.x and resolver 1.6. I've been running it on my Jenkins nodes for about a year now with great success. Before it, we had corrupted local maven repos (zip file empty kind of exceptions) a couple of times a week. Then, we configured Jenkins to properly handle multiple concurrent builds by not sharing the maven local repo and run each build in docker which is wasteful since you have to redownload all your dependencies on each build and build time is obviously very long.
Then I bumped into the lock feature which is available on resolver 1.6 (instructions here https://svn.apache.org/repos/asf/maven/website/components/resolver-archives/resolver-1.6.3/maven-resolver-synccontext-redisson/index.html). It is a bit cumbersome to setup but after that, we started sharing the same local repos and our build time dramatically improved (something like from 4 minutes to ~30 seconds) and we had a corrupted maven repo once or twice throughout the year (and it could have been some job that a dev created without the proper config, I didn't investigate much).
Redis is really easy to setup and is available via docker or apt so it should be a breeze to add it to your init script like we did. My nodes had docker already installed so it was just adding this to the "Init script" in the nodes configuration :
#!/bin/bash -ue
docker run -d --network host --name mvn_redis redis:6.2-alpine
I'd be interested to keep up with all the latest work that have been done on this in resolver 1.7, that's why I created another issue to request a java 8 maven 3.9 release.
On 2021/10/04 20:13:45, Francois Marot <fr...@gmail.com> wrote:
> "Salut" Michael,
>
> thanks for the detailled answer.
> Regarding my Jenkins 'hack' you can totally forget about it: it just
> ensures no more than one Maven execution uses a given repo at once by
> locating the local repositories in a folder reflecting the executor (which
> are kinda Jenkins "threads").
> I plan not to use your work right now mostly because it would involve
> setting up a Redis Database and my 'hack' still works (despite using a lot
> of disk space and being a bit slow).
>
> If I understand correctly after Tams' message, your work is mostly geared
> toward running multiple Maven build in parallel and on multiple hosts, all
> sharing one big "local" repository through the network. Am I correct ? In
> my case, I have multiple Jenkins workers but never thought about making
> them collaborate on a shared "local" repo. I'll have to think about it !
>
> Thanks !
>
> François
>
>
>
> Le lun. 4 oct. 2021 à 21:22, Michael Osipov <mi...@apache.org> a écrit :
>
> > Am 2021-10-04 um 14:31 schrieb Francois Marot:
> > > Hello all,
> > >
> > > I would like clarifications on MNG-2802[*] that seems to be solved. If I
> > > understand correctly it solves the problem where multiple simultaneous
> > > Maven executions shared the same local repository.
> > > I have been keeping for years a bit of code in my Jenkinsfiles ensuring
> > the
> > > local repos were accessed only by a single jenkins executor at the same
> > > time, so it seems like I can get rid of this doesn't it ?
> > > I'd like your input on this because reproducing the bug is difficult so
> > I'd
> > > like to be sure before simplifying my Jenkinsfile.
> > >
> > > Moreover, does anyone know how this problem is solved technically ? Using
> > > files lock ? Can anybody explain ?
> > >
> > > Thanks you and thanks the Maven team for keeping up the good work at such
> > > pace !
> >
> > Salut François,
> >
> > I don't know what you exactly fiddled in your Jenkinsfile, but there is
> > no way make it right just with Jenkinsfile foo when you want access
> > *one* local repo with *multiple* processes you *must* use an external
> > coordinator process.
> > Forget also about file locks, they are all advisory, at least on
> > POSIX-like filesystems. There is no way, or a very hard way to make them
> > right. We tried and removed them.
> >
> > Now, let me give you bit of history: I have integrated/implemented a
> > fully working multi-process Redisson-based synchronization lock factory
> > for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already
> > been battle tested for several months by a user on a busy CI server).
> > Tamás took it to the next level in Resolver 1.7.x and generalized it to
> > a named lock approach where the Redisson Lock Factory plugs in nicely
> > with other lock implementations. Almost nine months of work and it works
> > very good now.
> > Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x.
> > I have created a branch which reconciles best of breed: Maven 3.8.x and
> > Maven Resolver 1.7.x. It just works (also verified by this user). I also
> > ran more than a thousand builds with 8 parallel processes and 4 threads.
> > It will easily handle thousands of locks in parralel with an one digit
> > percent overhead.
> >
> > I don't know for what documentation you are waiting for, but everything
> > is there, you can start *now* to leverage it: [2], [3]. With Resolver
> > 2.0.0 and Maven 4.0.0 it will likely be even easier to integrate.
> >
> > Hazelcast will also work, but Tamás is the special here.
> >
> > Again: You *must* use an external process to synchronize access. No
> > excuses.
> >
> > Michael
> >
> > [1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x
> > [2]
> > https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
> > [3]
> >
> > https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> > For additional commands, e-mail: users-help@maven.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Francois Marot <fr...@gmail.com>.
"Salut" Michael,
thanks for the detailled answer.
Regarding my Jenkins 'hack' you can totally forget about it: it just
ensures no more than one Maven execution uses a given repo at once by
locating the local repositories in a folder reflecting the executor (which
are kinda Jenkins "threads").
I plan not to use your work right now mostly because it would involve
setting up a Redis Database and my 'hack' still works (despite using a lot
of disk space and being a bit slow).
If I understand correctly after Tams' message, your work is mostly geared
toward running multiple Maven build in parallel and on multiple hosts, all
sharing one big "local" repository through the network. Am I correct ? In
my case, I have multiple Jenkins workers but never thought about making
them collaborate on a shared "local" repo. I'll have to think about it !
Thanks !
François
Le lun. 4 oct. 2021 à 21:22, Michael Osipov <mi...@apache.org> a écrit :
> Am 2021-10-04 um 14:31 schrieb Francois Marot:
> > Hello all,
> >
> > I would like clarifications on MNG-2802[*] that seems to be solved. If I
> > understand correctly it solves the problem where multiple simultaneous
> > Maven executions shared the same local repository.
> > I have been keeping for years a bit of code in my Jenkinsfiles ensuring
> the
> > local repos were accessed only by a single jenkins executor at the same
> > time, so it seems like I can get rid of this doesn't it ?
> > I'd like your input on this because reproducing the bug is difficult so
> I'd
> > like to be sure before simplifying my Jenkinsfile.
> >
> > Moreover, does anyone know how this problem is solved technically ? Using
> > files lock ? Can anybody explain ?
> >
> > Thanks you and thanks the Maven team for keeping up the good work at such
> > pace !
>
> Salut François,
>
> I don't know what you exactly fiddled in your Jenkinsfile, but there is
> no way make it right just with Jenkinsfile foo when you want access
> *one* local repo with *multiple* processes you *must* use an external
> coordinator process.
> Forget also about file locks, they are all advisory, at least on
> POSIX-like filesystems. There is no way, or a very hard way to make them
> right. We tried and removed them.
>
> Now, let me give you bit of history: I have integrated/implemented a
> fully working multi-process Redisson-based synchronization lock factory
> for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already
> been battle tested for several months by a user on a busy CI server).
> Tamás took it to the next level in Resolver 1.7.x and generalized it to
> a named lock approach where the Redisson Lock Factory plugs in nicely
> with other lock implementations. Almost nine months of work and it works
> very good now.
> Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x.
> I have created a branch which reconciles best of breed: Maven 3.8.x and
> Maven Resolver 1.7.x. It just works (also verified by this user). I also
> ran more than a thousand builds with 8 parallel processes and 4 threads.
> It will easily handle thousands of locks in parralel with an one digit
> percent overhead.
>
> I don't know for what documentation you are waiting for, but everything
> is there, you can start *now* to leverage it: [2], [3]. With Resolver
> 2.0.0 and Maven 4.0.0 it will likely be even easier to integrate.
>
> Hazelcast will also work, but Tamás is the special here.
>
> Again: You *must* use an external process to synchronize access. No
> excuses.
>
> Michael
>
> [1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x
> [2]
> https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
> [3]
>
> https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Michael Osipov <mi...@apache.org>.
Am 2021-10-04 um 14:31 schrieb Francois Marot:
> Hello all,
>
> I would like clarifications on MNG-2802[*] that seems to be solved. If I
> understand correctly it solves the problem where multiple simultaneous
> Maven executions shared the same local repository.
> I have been keeping for years a bit of code in my Jenkinsfiles ensuring the
> local repos were accessed only by a single jenkins executor at the same
> time, so it seems like I can get rid of this doesn't it ?
> I'd like your input on this because reproducing the bug is difficult so I'd
> like to be sure before simplifying my Jenkinsfile.
>
> Moreover, does anyone know how this problem is solved technically ? Using
> files lock ? Can anybody explain ?
>
> Thanks you and thanks the Maven team for keeping up the good work at such
> pace !
Salut François,
I don't know what you exactly fiddled in your Jenkinsfile, but there is
no way make it right just with Jenkinsfile foo when you want access
*one* local repo with *multiple* processes you *must* use an external
coordinator process.
Forget also about file locks, they are all advisory, at least on
POSIX-like filesystems. There is no way, or a very hard way to make them
right. We tried and removed them.
Now, let me give you bit of history: I have integrated/implemented a
fully working multi-process Redisson-based synchronization lock factory
for Maven Resolver 1.6.x which ships with Maven 3.8.x. (It has already
been battle tested for several months by a user on a busy CI server).
Tamás took it to the next level in Resolver 1.7.x and generalized it to
a named lock approach where the Redisson Lock Factory plugs in nicely
with other lock implementations. Almost nine months of work and it works
very good now.
Now here is the problem: Maven Resolver 1.7.x comes only with Maven 4.x.
I have created a branch which reconciles best of breed: Maven 3.8.x and
Maven Resolver 1.7.x. It just works (also verified by this user). I also
ran more than a thousand builds with 8 parallel processes and 4 threads.
It will easily handle thousands of locks in parralel with an one digit
percent overhead.
I don't know for what documentation you are waiting for, but everything
is there, you can start *now* to leverage it: [2], [3]. With Resolver
2.0.0 and Maven 4.0.0 it will likely be even easier to integrate.
Hazelcast will also work, but Tamás is the special here.
Again: You *must* use an external process to synchronize access. No excuses.
Michael
[1] https://github.com/apache/maven/tree/maven-3.8.x-resolver-1.7.x
[2] https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
[3]
https://maven.apache.org/resolver/maven-resolver-named-locks-redisson/index.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Francois Marot <fr...@gmail.com>.
Wooohaa, thanks Thomas.
It's a whole new world being revealed to me: I wasn't aware of the huge
work happening on artifact resolvers.
At the same time, it seems not to be available out of the box (needs a
Redis server) so I'll keep my Jenkinsfile hacks for now and will wait
for proper documentation and how-to.
Thanks for the pointers !
*François Marot*
Le lun. 4 oct. 2021 à 15:10, Thomas Broyer <t....@gmail.com> a écrit :
> From JIRA links, this is apparently fixed by
> https://issues.apache.org/jira/browse/MRESOLVER-131, which means you need
> to deploy a Redis server.
> There apparently also is an Hazelcast-based version:
> https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
>
> On Mon, Oct 4, 2021 at 2:32 PM Francois Marot <fr...@gmail.com>
> wrote:
>
> > Hello all,
> >
> > I would like clarifications on MNG-2802[*] that seems to be solved. If I
> > understand correctly it solves the problem where multiple simultaneous
> > Maven executions shared the same local repository.
> > I have been keeping for years a bit of code in my Jenkinsfiles ensuring
> the
> > local repos were accessed only by a single jenkins executor at the same
> > time, so it seems like I can get rid of this doesn't it ?
> > I'd like your input on this because reproducing the bug is difficult so
> I'd
> > like to be sure before simplifying my Jenkinsfile.
> >
> > Moreover, does anyone know how this problem is solved technically ? Using
> > files lock ? Can anybody explain ?
> >
> > Thanks you and thanks the Maven team for keeping up the good work at such
> > pace !
> >
> > François MAROT
> >
> >
> > * https://issues.apache.org/jira/browse/MNG-2802
> >
>
>
> --
> Thomas Broyer
> /tɔ.ma.bʁwa.je/ <http://xn--nna.ma.xn--bwa-xxb.je/> <
> http://xn--nna.ma.xn--bwa-xxb.je/>
>
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Oliver Fischer <o....@swe-blog.net>.
Yes, these are good news!
Am 04.10.21 um 15:10 schrieb Thomas Broyer:
> From JIRA links, this is apparently fixed by
> https://issues.apache.org/jira/browse/MRESOLVER-131, which means you need
> to deploy a Redis server.
> There apparently also is an Hazelcast-based version:
> https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
>
> On Mon, Oct 4, 2021 at 2:32 PM Francois Marot <fr...@gmail.com>
> wrote:
>
>> Hello all,
>>
>> I would like clarifications on MNG-2802[*] that seems to be solved. If I
>> understand correctly it solves the problem where multiple simultaneous
>> Maven executions shared the same local repository.
>> I have been keeping for years a bit of code in my Jenkinsfiles ensuring the
>> local repos were accessed only by a single jenkins executor at the same
>> time, so it seems like I can get rid of this doesn't it ?
>> I'd like your input on this because reproducing the bug is difficult so I'd
>> like to be sure before simplifying my Jenkinsfile.
>>
>> Moreover, does anyone know how this problem is solved technically ? Using
>> files lock ? Can anybody explain ?
>>
>> Thanks you and thanks the Maven team for keeping up the good work at such
>> pace !
>>
>> François MAROT
>>
>>
>> * https://issues.apache.org/jira/browse/MNG-2802
>>
>
--
N Oliver B. Fischer
A Schönhauser Allee 64, 10437 Berlin, Deutschland/Germany
P +49 30 44793251
M +49 178 7903538
E o.b.fischer@swe-blog.net
S oliver.b.fischer
J oliver.b.fischer@jabber.org
X http://xing.to/obf
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Local repository accessed by multiple Maven instances - how is it
solved ?
Posted by Thomas Broyer <t....@gmail.com>.
From JIRA links, this is apparently fixed by
https://issues.apache.org/jira/browse/MRESOLVER-131, which means you need
to deploy a Redis server.
There apparently also is an Hazelcast-based version:
https://maven.apache.org/resolver/maven-resolver-named-locks/index.html
On Mon, Oct 4, 2021 at 2:32 PM Francois Marot <fr...@gmail.com>
wrote:
> Hello all,
>
> I would like clarifications on MNG-2802[*] that seems to be solved. If I
> understand correctly it solves the problem where multiple simultaneous
> Maven executions shared the same local repository.
> I have been keeping for years a bit of code in my Jenkinsfiles ensuring the
> local repos were accessed only by a single jenkins executor at the same
> time, so it seems like I can get rid of this doesn't it ?
> I'd like your input on this because reproducing the bug is difficult so I'd
> like to be sure before simplifying my Jenkinsfile.
>
> Moreover, does anyone know how this problem is solved technically ? Using
> files lock ? Can anybody explain ?
>
> Thanks you and thanks the Maven team for keeping up the good work at such
> pace !
>
> François MAROT
>
>
> * https://issues.apache.org/jira/browse/MNG-2802
>
--
Thomas Broyer
/tɔ.ma.bʁwa.je/ <http://xn--nna.ma.xn--bwa-xxb.je/>