You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Dan Tran <da...@gmail.com> on 2021/05/02 20:24:20 UTC
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
we are in pain with this issue :-)
-D
On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana <ti...@apache.org> wrote:
> Hi Falco,
>
> This is not the first time I have been talking about these principles in
> our team.
> Seven years ago and then in 2019. But sorry I cannot force the people to do
> it and they have to start by themself. We have to do it together.
> All I can do is to provide some training and elaborate a problem but I am
> convinced that we will meet again after the next seven years if we
> don't understand the JMM. Thus we should prevent from redoing the same bad
> and understand the process on how to make the code better in the early
> beginning.
>
> Cheers
> Tibor17
>
>
> On Sun, Jan 24, 2021 at 10:51 PM Falko Modler <f....@gmx.net> wrote:
>
> > Hi Tibor,
> >
> > thanks for this very elaborate answer and I always appreciate your
> > feedback, but to me it kind of misses the point a bit...?
> >
> > > may not necessarily have to do with concurrent access.
> > But it does in this special case. Please see the issue and the linked
> > explanations.
> >
> > > The solution with ThreadLocal would eat too much memory.
> > Is that so? Are you sure about this? How much is "too much"?
> > Are there any predefined profiling tests I can run?
> >
> > I mean: yes, it is a workaround and immutable core classes that are
> > _designed_ for concurrent access would be much better,
> > but who is going to do such a massive refactoring (without breaking
> > Maven extensions that are today mutating MavenProject etc.)?
> >
> > TBH, this is one of the, IMHO, critical bugs that should have been fixed
> > before Maven 4.
> >
> > Cheers,
> > Falko
> >
> > Am 21.01.2021 um 02:13 schrieb Tibor Digana:
> > > I commented on one issue regarding the NULL JAR file in Artifact a few
> > days
> > > ago.
> > > The thing that some data is "missing" in some large object structures
> in
> > > the environment with multiple threads may not necessarily have to do
> with
> > > concurrent access.
> > > There may not be any writes to MavenProject or MavenSession causing
> > > "missed" data, and the answer why this happens is Memory Model.
> > >
> > > It's the fact that non-concurrent or non-immutable objects may lose
> some
> > > references very easily!
> > > This has all to do with JMM and not the happens-before relationship.
> > >
> > > Suppose that we have thread T1 creating ArrayList and adding elements
> > into
> > > this collection.
> > > artifacts = new ArrayList();
> > > artifacts.add(new DefaultArtifact(...));
> > >
> > > Suppose thread T2 reads the artifacts from the collection right after
> > > "artifacts.add()".
> > > Artifact a = artifacts.get(0);
> > >
> > > In practice the following happens:
> > > artifacts.size() returns 1
> > > but artifacts.get(0) returns NULL
> > >
> > > Let;s explain why it happens.
> > > The implementation of ArrayList is not native. It is a pure Java
> > > implementation which has two variables inside:
> > > + count:int
> > > + array:Object[]
> > > These two variables always appear in a critical section and they do not
> > > have proper treatments in ArrayList.
> > > Technically, the things are complicated on the CPU level and more
> > > complicated than happens-before theorems.
> > > T1 contains pointers and data in CPU registers or CPU cache. No Thread
> > has
> > > a direct access to a stack of another Thread, and of course it does not
> > > operate on main memory.
> > > The CPU uses memory barriers (assembler instructions) and a cache to
> > > operate with RAM and memory coherency.
> > > These instructions are used via Java keywords: final, volatile and
> > > synchronized.
> > > T2 may not see all elements completely from the ArrayList because there
> > are
> > > no safety mechanisms in the implementation of ArrayList to make this
> > happen.
> > > Thus the T2 may see the values in the Java variable "count" *but it may
> > not
> > > see the values in* "array", or vice versa.
> > >
> > > The results are NPE, or missing JAR artifacts or the issues with Maven
> > > Resolver, as we can see in https://github.com/apache/maven/pull/310
> > >
> > > The solution with ThreadLocal would eat too much memory.
> > > Reimplementing the POJO classes in Maven and making them thread safe
> > would
> > > solve many issues in the Core and Resolver.
> > > Considering my examples with ArrayList, the thread safety should
> continue
> > > deeper with the implementation of DefaultArtifact, etc.
> > > In my experience, it's worth using the collection which appears in the
> > > package "java.util.concurrent".
> > > For instance, I use ConcurrentLinkedDequeue for simple iterators with
> > small
> > > amounts of elements. Alternatively use COWAL for large data and
> > reordering
> > > of elements by adding or removing them somewhere inside.
> > >
> > > Cheers
> > > Tibor17
> > >
> > >
> > >
> > > On Sat, Jan 16, 2021 at 10:21 PM Dan Tran <da...@gmail.com> wrote:
> > >
> > >> we are facing the same issue at work (300+ modules), classpath
> > >> empty randomly empty
> > >>
> > >> Love to see some resolution, will help to test it
> > >>
> > >> Thanks
> > >>
> > >> -D
> > >>
> > >> On Fri, Jan 15, 2021 at 1:51 PM Falko Modler <f....@gmx.net>
> wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> I'd like to raise awareness for the MavenProject concurrency problem
> > >>> that is causing MNG-6843 "Parallel build fails due to missing JAR
> > >>> artifacts in compilePath" [1] and probably others [2] [3] [4].
> > >>>
> > >>> Almost a month ago, I created a ThreadLocal-based fix for this [5]
> > >>> (after another, older cloning-based approach had raised some concerns
> > by
> > >>> Robert Scholte [6]).
> > >>>
> > >>> Michael Osipov was the only one so far having a look (thanks!) and he
> > >>> suggested that more Maven team members should review this.
> > >>>
> > >>> So, before I take a stab at the not so trivial integration test that
> > >>> Michael proposed [7], I'd like to get an approval for the general
> > >>> aproach (or a declination in case someone has a better idea).
> > >>>
> > >>> Thanks for your attention and feedback!
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Falko
> > >>>
> > >>>
> > >>> [1] https://issues.apache.org/jira/browse/MNG-6843
> > >>>
> > >>> [2] https://issues.apache.org/jira/browse/MNG-4996
> > >>>
> > >>> [3] https://issues.apache.org/jira/browse/MNG-5750
> > >>>
> > >>> [4] https://issues.apache.org/jira/browse/MNG-5960
> > >>>
> > >>> [5] https://github.com/apache/maven/pull/413
> > >>>
> > >>> [6] https://github.com/apache/maven/pull/310#issuecomment-571317501
> > >>>
> > >>> [7] https://github.com/apache/maven/pull/413#issuecomment-754661032
> > >>>
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>> For additional commands, e-mail: dev-help@maven.apache.org
> > >>>
> > >>>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
Thanks, will hook it up with our internal build
On Sun, Jul 11, 2021 at 6:59 PM Olivier Lamy <ol...@apache.org> wrote:
> done
>
> On Sun, 11 Jul 2021 at 07:44, Dan Tran <da...@gmail.com> wrote:
>
> > possible to push the latest snapshot to
> >
> >
> https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven
> > ?
> >
> > Thanks
> >
> > -D
> >
> > On Sat, Jul 10, 2021 at 2:33 PM Dan Tran <da...@gmail.com> wrote:
> >
> > > Thanks, I will test out the latest 3.8
> > >
> > > -D
> > >
> > > On Sat, Jul 10, 2021 at 12:29 PM Falko Modler <f....@gmx.net>
> wrote:
> > >
> > >> Done already: https://github.com/apache/maven/pull/482
> > >>
> > >> Cheers,
> > >> Falko
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >> For additional commands, e-mail: dev-help@maven.apache.org
> > >>
> > >>
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Olivier Lamy <ol...@apache.org>.
done
On Sun, 11 Jul 2021 at 07:44, Dan Tran <da...@gmail.com> wrote:
> possible to push the latest snapshot to
>
> https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven
> ?
>
> Thanks
>
> -D
>
> On Sat, Jul 10, 2021 at 2:33 PM Dan Tran <da...@gmail.com> wrote:
>
> > Thanks, I will test out the latest 3.8
> >
> > -D
> >
> > On Sat, Jul 10, 2021 at 12:29 PM Falko Modler <f....@gmx.net> wrote:
> >
> >> Done already: https://github.com/apache/maven/pull/482
> >>
> >> Cheers,
> >> Falko
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >> For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >>
>
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
possible to push the latest snapshot to
https://repository.apache.org/content/groups/snapshots/org/apache/maven/apache-maven
?
Thanks
-D
On Sat, Jul 10, 2021 at 2:33 PM Dan Tran <da...@gmail.com> wrote:
> Thanks, I will test out the latest 3.8
>
> -D
>
> On Sat, Jul 10, 2021 at 12:29 PM Falko Modler <f....@gmx.net> wrote:
>
>> Done already: https://github.com/apache/maven/pull/482
>>
>> Cheers,
>> Falko
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
Thanks, I will test out the latest 3.8
-D
On Sat, Jul 10, 2021 at 12:29 PM Falko Modler <f....@gmx.net> wrote:
> Done already: https://github.com/apache/maven/pull/482
>
> Cheers,
> Falko
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in
compilePath
Posted by Falko Modler <f....@gmx.net>.
Done already: https://github.com/apache/maven/pull/482
Cheers,
Falko
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
I think this is the one https://github.com/apache/maven/pull/413
-D
On Sat, Jul 10, 2021 at 2:56 AM Michael Osipov <mi...@apache.org> wrote:
> Am 2021-07-10 um 00:52 schrieb Dan Tran:
> > ah sorry, the proposed PR merged into master. Can we back port it to
> 3.8?
>
> Which PR exactly?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in
compilePath
Posted by Michael Osipov <mi...@apache.org>.
Am 2021-07-10 um 00:52 schrieb Dan Tran:
> ah sorry, the proposed PR merged into master. Can we back port it to 3.8?
Which PR exactly?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
ah sorry, the proposed PR merged into master. Can we back port it to 3.8?
-D
On Fri, Jul 9, 2021 at 3:47 PM Dan Tran <da...@gmail.com> wrote:
> ping again :-) to keep discussion going :-)
>
> -D
>
> On Tue, May 18, 2021 at 6:20 AM Guillaume Nodet <gn...@apache.org> wrote:
>
>> Maybe not that many people are using parallel builds...
>> Imho, the PR should be merged. I've created
>> https://issues.apache.org/jira/browse/MNG-7157 to provide a better API
>> and deprecate the getArtifacts() method which is flawed (see discussion on
>> the PR https://github.com/apache/maven/pull/413#issuecomment-841720517).
>>
>> Le mar. 4 mai 2021 à 15:15, Michael Osipov <mi...@apache.org> a écrit
>> :
>>
>> > Am 2021-05-03 um 22:15 schrieb Falko Modler:
>> > > Earlier today Michael Osipov added this note to the ticket:
>> > >
>> > > I confirm that this still happens on Maven master with Resolver
>> > > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency
>> > > resolution happens in the build.
>> >
>> > I am still confused why only a few suffer from this. Is this related to
>> > dependency inheritance?
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> > For additional commands, e-mail: dev-help@maven.apache.org
>> >
>> >
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Dan Tran <da...@gmail.com>.
ping again :-) to keep discussion going :-)
-D
On Tue, May 18, 2021 at 6:20 AM Guillaume Nodet <gn...@apache.org> wrote:
> Maybe not that many people are using parallel builds...
> Imho, the PR should be merged. I've created
> https://issues.apache.org/jira/browse/MNG-7157 to provide a better API
> and deprecate the getArtifacts() method which is flawed (see discussion on
> the PR https://github.com/apache/maven/pull/413#issuecomment-841720517).
>
> Le mar. 4 mai 2021 à 15:15, Michael Osipov <mi...@apache.org> a écrit :
>
> > Am 2021-05-03 um 22:15 schrieb Falko Modler:
> > > Earlier today Michael Osipov added this note to the ticket:
> > >
> > > I confirm that this still happens on Maven master with Resolver
> > > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency
> > > resolution happens in the build.
> >
> > I am still confused why only a few suffer from this. Is this related to
> > dependency inheritance?
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > For additional commands, e-mail: dev-help@maven.apache.org
> >
> >
>
> --
> ------------------------
> Guillaume Nodet
>
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Guillaume Nodet <gn...@apache.org>.
Maybe not that many people are using parallel builds...
Imho, the PR should be merged. I've created
https://issues.apache.org/jira/browse/MNG-7157 to provide a better API
and deprecate the getArtifacts() method which is flawed (see discussion on
the PR https://github.com/apache/maven/pull/413#issuecomment-841720517).
Le mar. 4 mai 2021 à 15:15, Michael Osipov <mi...@apache.org> a écrit :
> Am 2021-05-03 um 22:15 schrieb Falko Modler:
> > Earlier today Michael Osipov added this note to the ticket:
> >
> > I confirm that this still happens on Maven master with Resolver
> > 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency
> > resolution happens in the build.
>
> I am still confused why only a few suffer from this. Is this related to
> dependency inheritance?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
------------------------
Guillaume Nodet
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in
compilePath
Posted by Michael Osipov <mi...@apache.org>.
Am 2021-05-03 um 22:15 schrieb Falko Modler:
> Earlier today Michael Osipov added this note to the ticket:
>
> I confirm that this still happens on Maven master with Resolver
> 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency
> resolution happens in the build.
I am still confused why only a few suffer from this. Is this related to
dependency inheritance?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in
compilePath
Posted by Falko Modler <f....@gmx.net>.
Earlier today Michael Osipov added this note to the ticket:
I confirm that this still happens on Maven master with Resolver 1.7.0-SNAPSHOT. This isn't Resolver related because no dependency resolution happens in the build.
Cheers,
Falko
Am 03.05.2021 um 02:25 schrieb Tibor Digana:
> We have already moved ahead and the new Resolver should solve this issue.
> This looks similar to the issue in Surefire which was closed as a bug of
> Resoler. Michael is currently working on a new Resolver and he was
> participating in the analysis of the bug too. So, maybe it is still the
> same root cause for this one.
>
> T
>
> On Sun, May 2, 2021 at 10:24 PM Dan Tran <da...@gmail.com> wrote:
>
>> we are in pain with this issue :-)
>>
>> -D
>>
>> On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana <ti...@apache.org>
>> wrote:
>>
>>> Hi Falco,
>>>
>>> This is not the first time I have been talking about these principles in
>>> our team.
>>> Seven years ago and then in 2019. But sorry I cannot force the people to
>> do
>>> it and they have to start by themself. We have to do it together.
>>> All I can do is to provide some training and elaborate a problem but I am
>>> convinced that we will meet again after the next seven years if we
>>> don't understand the JMM. Thus we should prevent from redoing the same
>> bad
>>> and understand the process on how to make the code better in the early
>>> beginning.
>>>
>>> Cheers
>>> Tibor17
>>>
>>>
>>> On Sun, Jan 24, 2021 at 10:51 PM Falko Modler <f....@gmx.net> wrote:
>>>
>>>> Hi Tibor,
>>>>
>>>> thanks for this very elaborate answer and I always appreciate your
>>>> feedback, but to me it kind of misses the point a bit...?
>>>>
>>>>> may not necessarily have to do with concurrent access.
>>>> But it does in this special case. Please see the issue and the linked
>>>> explanations.
>>>>
>>>>> The solution with ThreadLocal would eat too much memory.
>>>> Is that so? Are you sure about this? How much is "too much"?
>>>> Are there any predefined profiling tests I can run?
>>>>
>>>> I mean: yes, it is a workaround and immutable core classes that are
>>>> _designed_ for concurrent access would be much better,
>>>> but who is going to do such a massive refactoring (without breaking
>>>> Maven extensions that are today mutating MavenProject etc.)?
>>>>
>>>> TBH, this is one of the, IMHO, critical bugs that should have been
>> fixed
>>>> before Maven 4.
>>>>
>>>> Cheers,
>>>> Falko
>>>>
>>>> Am 21.01.2021 um 02:13 schrieb Tibor Digana:
>>>>> I commented on one issue regarding the NULL JAR file in Artifact a
>> few
>>>> days
>>>>> ago.
>>>>> The thing that some data is "missing" in some large object structures
>>> in
>>>>> the environment with multiple threads may not necessarily have to do
>>> with
>>>>> concurrent access.
>>>>> There may not be any writes to MavenProject or MavenSession causing
>>>>> "missed" data, and the answer why this happens is Memory Model.
>>>>>
>>>>> It's the fact that non-concurrent or non-immutable objects may lose
>>> some
>>>>> references very easily!
>>>>> This has all to do with JMM and not the happens-before relationship.
>>>>>
>>>>> Suppose that we have thread T1 creating ArrayList and adding elements
>>>> into
>>>>> this collection.
>>>>> artifacts = new ArrayList();
>>>>> artifacts.add(new DefaultArtifact(...));
>>>>>
>>>>> Suppose thread T2 reads the artifacts from the collection right after
>>>>> "artifacts.add()".
>>>>> Artifact a = artifacts.get(0);
>>>>>
>>>>> In practice the following happens:
>>>>> artifacts.size() returns 1
>>>>> but artifacts.get(0) returns NULL
>>>>>
>>>>> Let;s explain why it happens.
>>>>> The implementation of ArrayList is not native. It is a pure Java
>>>>> implementation which has two variables inside:
>>>>> + count:int
>>>>> + array:Object[]
>>>>> These two variables always appear in a critical section and they do
>> not
>>>>> have proper treatments in ArrayList.
>>>>> Technically, the things are complicated on the CPU level and more
>>>>> complicated than happens-before theorems.
>>>>> T1 contains pointers and data in CPU registers or CPU cache. No
>> Thread
>>>> has
>>>>> a direct access to a stack of another Thread, and of course it does
>> not
>>>>> operate on main memory.
>>>>> The CPU uses memory barriers (assembler instructions) and a cache to
>>>>> operate with RAM and memory coherency.
>>>>> These instructions are used via Java keywords: final, volatile and
>>>>> synchronized.
>>>>> T2 may not see all elements completely from the ArrayList because
>> there
>>>> are
>>>>> no safety mechanisms in the implementation of ArrayList to make this
>>>> happen.
>>>>> Thus the T2 may see the values in the Java variable "count" *but it
>> may
>>>> not
>>>>> see the values in* "array", or vice versa.
>>>>>
>>>>> The results are NPE, or missing JAR artifacts or the issues with
>> Maven
>>>>> Resolver, as we can see in https://github.com/apache/maven/pull/310
>>>>>
>>>>> The solution with ThreadLocal would eat too much memory.
>>>>> Reimplementing the POJO classes in Maven and making them thread safe
>>>> would
>>>>> solve many issues in the Core and Resolver.
>>>>> Considering my examples with ArrayList, the thread safety should
>>> continue
>>>>> deeper with the implementation of DefaultArtifact, etc.
>>>>> In my experience, it's worth using the collection which appears in
>> the
>>>>> package "java.util.concurrent".
>>>>> For instance, I use ConcurrentLinkedDequeue for simple iterators with
>>>> small
>>>>> amounts of elements. Alternatively use COWAL for large data and
>>>> reordering
>>>>> of elements by adding or removing them somewhere inside.
>>>>>
>>>>> Cheers
>>>>> Tibor17
>>>>>
>>>>>
>>>>>
>>>>> On Sat, Jan 16, 2021 at 10:21 PM Dan Tran <da...@gmail.com> wrote:
>>>>>
>>>>>> we are facing the same issue at work (300+ modules), classpath
>>>>>> empty randomly empty
>>>>>>
>>>>>> Love to see some resolution, will help to test it
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> -D
>>>>>>
>>>>>> On Fri, Jan 15, 2021 at 1:51 PM Falko Modler <f....@gmx.net>
>>> wrote:
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I'd like to raise awareness for the MavenProject concurrency
>> problem
>>>>>>> that is causing MNG-6843 "Parallel build fails due to missing JAR
>>>>>>> artifacts in compilePath" [1] and probably others [2] [3] [4].
>>>>>>>
>>>>>>> Almost a month ago, I created a ThreadLocal-based fix for this [5]
>>>>>>> (after another, older cloning-based approach had raised some
>> concerns
>>>> by
>>>>>>> Robert Scholte [6]).
>>>>>>>
>>>>>>> Michael Osipov was the only one so far having a look (thanks!) and
>> he
>>>>>>> suggested that more Maven team members should review this.
>>>>>>>
>>>>>>> So, before I take a stab at the not so trivial integration test
>> that
>>>>>>> Michael proposed [7], I'd like to get an approval for the general
>>>>>>> aproach (or a declination in case someone has a better idea).
>>>>>>>
>>>>>>> Thanks for your attention and feedback!
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Falko
>>>>>>>
>>>>>>>
>>>>>>> [1] https://issues.apache.org/jira/browse/MNG-6843
>>>>>>>
>>>>>>> [2] https://issues.apache.org/jira/browse/MNG-4996
>>>>>>>
>>>>>>> [3] https://issues.apache.org/jira/browse/MNG-5750
>>>>>>>
>>>>>>> [4] https://issues.apache.org/jira/browse/MNG-5960
>>>>>>>
>>>>>>> [5] https://github.com/apache/maven/pull/413
>>>>>>>
>>>>>>> [6]
>> https://github.com/apache/maven/pull/310#issuecomment-571317501
>>>>>>> [7]
>> https://github.com/apache/maven/pull/413#issuecomment-754661032
>>>>>>>
>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>>>>
>>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>>
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: MNG-6843 Parallel build fails due to missing JAR artifacts in compilePath
Posted by Tibor Digana <ti...@apache.org>.
We have already moved ahead and the new Resolver should solve this issue.
This looks similar to the issue in Surefire which was closed as a bug of
Resoler. Michael is currently working on a new Resolver and he was
participating in the analysis of the bug too. So, maybe it is still the
same root cause for this one.
T
On Sun, May 2, 2021 at 10:24 PM Dan Tran <da...@gmail.com> wrote:
> we are in pain with this issue :-)
>
> -D
>
> On Sun, Jan 24, 2021 at 2:56 PM Tibor Digana <ti...@apache.org>
> wrote:
>
> > Hi Falco,
> >
> > This is not the first time I have been talking about these principles in
> > our team.
> > Seven years ago and then in 2019. But sorry I cannot force the people to
> do
> > it and they have to start by themself. We have to do it together.
> > All I can do is to provide some training and elaborate a problem but I am
> > convinced that we will meet again after the next seven years if we
> > don't understand the JMM. Thus we should prevent from redoing the same
> bad
> > and understand the process on how to make the code better in the early
> > beginning.
> >
> > Cheers
> > Tibor17
> >
> >
> > On Sun, Jan 24, 2021 at 10:51 PM Falko Modler <f....@gmx.net> wrote:
> >
> > > Hi Tibor,
> > >
> > > thanks for this very elaborate answer and I always appreciate your
> > > feedback, but to me it kind of misses the point a bit...?
> > >
> > > > may not necessarily have to do with concurrent access.
> > > But it does in this special case. Please see the issue and the linked
> > > explanations.
> > >
> > > > The solution with ThreadLocal would eat too much memory.
> > > Is that so? Are you sure about this? How much is "too much"?
> > > Are there any predefined profiling tests I can run?
> > >
> > > I mean: yes, it is a workaround and immutable core classes that are
> > > _designed_ for concurrent access would be much better,
> > > but who is going to do such a massive refactoring (without breaking
> > > Maven extensions that are today mutating MavenProject etc.)?
> > >
> > > TBH, this is one of the, IMHO, critical bugs that should have been
> fixed
> > > before Maven 4.
> > >
> > > Cheers,
> > > Falko
> > >
> > > Am 21.01.2021 um 02:13 schrieb Tibor Digana:
> > > > I commented on one issue regarding the NULL JAR file in Artifact a
> few
> > > days
> > > > ago.
> > > > The thing that some data is "missing" in some large object structures
> > in
> > > > the environment with multiple threads may not necessarily have to do
> > with
> > > > concurrent access.
> > > > There may not be any writes to MavenProject or MavenSession causing
> > > > "missed" data, and the answer why this happens is Memory Model.
> > > >
> > > > It's the fact that non-concurrent or non-immutable objects may lose
> > some
> > > > references very easily!
> > > > This has all to do with JMM and not the happens-before relationship.
> > > >
> > > > Suppose that we have thread T1 creating ArrayList and adding elements
> > > into
> > > > this collection.
> > > > artifacts = new ArrayList();
> > > > artifacts.add(new DefaultArtifact(...));
> > > >
> > > > Suppose thread T2 reads the artifacts from the collection right after
> > > > "artifacts.add()".
> > > > Artifact a = artifacts.get(0);
> > > >
> > > > In practice the following happens:
> > > > artifacts.size() returns 1
> > > > but artifacts.get(0) returns NULL
> > > >
> > > > Let;s explain why it happens.
> > > > The implementation of ArrayList is not native. It is a pure Java
> > > > implementation which has two variables inside:
> > > > + count:int
> > > > + array:Object[]
> > > > These two variables always appear in a critical section and they do
> not
> > > > have proper treatments in ArrayList.
> > > > Technically, the things are complicated on the CPU level and more
> > > > complicated than happens-before theorems.
> > > > T1 contains pointers and data in CPU registers or CPU cache. No
> Thread
> > > has
> > > > a direct access to a stack of another Thread, and of course it does
> not
> > > > operate on main memory.
> > > > The CPU uses memory barriers (assembler instructions) and a cache to
> > > > operate with RAM and memory coherency.
> > > > These instructions are used via Java keywords: final, volatile and
> > > > synchronized.
> > > > T2 may not see all elements completely from the ArrayList because
> there
> > > are
> > > > no safety mechanisms in the implementation of ArrayList to make this
> > > happen.
> > > > Thus the T2 may see the values in the Java variable "count" *but it
> may
> > > not
> > > > see the values in* "array", or vice versa.
> > > >
> > > > The results are NPE, or missing JAR artifacts or the issues with
> Maven
> > > > Resolver, as we can see in https://github.com/apache/maven/pull/310
> > > >
> > > > The solution with ThreadLocal would eat too much memory.
> > > > Reimplementing the POJO classes in Maven and making them thread safe
> > > would
> > > > solve many issues in the Core and Resolver.
> > > > Considering my examples with ArrayList, the thread safety should
> > continue
> > > > deeper with the implementation of DefaultArtifact, etc.
> > > > In my experience, it's worth using the collection which appears in
> the
> > > > package "java.util.concurrent".
> > > > For instance, I use ConcurrentLinkedDequeue for simple iterators with
> > > small
> > > > amounts of elements. Alternatively use COWAL for large data and
> > > reordering
> > > > of elements by adding or removing them somewhere inside.
> > > >
> > > > Cheers
> > > > Tibor17
> > > >
> > > >
> > > >
> > > > On Sat, Jan 16, 2021 at 10:21 PM Dan Tran <da...@gmail.com> wrote:
> > > >
> > > >> we are facing the same issue at work (300+ modules), classpath
> > > >> empty randomly empty
> > > >>
> > > >> Love to see some resolution, will help to test it
> > > >>
> > > >> Thanks
> > > >>
> > > >> -D
> > > >>
> > > >> On Fri, Jan 15, 2021 at 1:51 PM Falko Modler <f....@gmx.net>
> > wrote:
> > > >>
> > > >>> Hi everyone,
> > > >>>
> > > >>> I'd like to raise awareness for the MavenProject concurrency
> problem
> > > >>> that is causing MNG-6843 "Parallel build fails due to missing JAR
> > > >>> artifacts in compilePath" [1] and probably others [2] [3] [4].
> > > >>>
> > > >>> Almost a month ago, I created a ThreadLocal-based fix for this [5]
> > > >>> (after another, older cloning-based approach had raised some
> concerns
> > > by
> > > >>> Robert Scholte [6]).
> > > >>>
> > > >>> Michael Osipov was the only one so far having a look (thanks!) and
> he
> > > >>> suggested that more Maven team members should review this.
> > > >>>
> > > >>> So, before I take a stab at the not so trivial integration test
> that
> > > >>> Michael proposed [7], I'd like to get an approval for the general
> > > >>> aproach (or a declination in case someone has a better idea).
> > > >>>
> > > >>> Thanks for your attention and feedback!
> > > >>>
> > > >>> Cheers,
> > > >>>
> > > >>> Falko
> > > >>>
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/MNG-6843
> > > >>>
> > > >>> [2] https://issues.apache.org/jira/browse/MNG-4996
> > > >>>
> > > >>> [3] https://issues.apache.org/jira/browse/MNG-5750
> > > >>>
> > > >>> [4] https://issues.apache.org/jira/browse/MNG-5960
> > > >>>
> > > >>> [5] https://github.com/apache/maven/pull/413
> > > >>>
> > > >>> [6]
> https://github.com/apache/maven/pull/310#issuecomment-571317501
> > > >>>
> > > >>> [7]
> https://github.com/apache/maven/pull/413#issuecomment-754661032
> > > >>>
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>> For additional commands, e-mail: dev-help@maven.apache.org
> > > >>>
> > > >>>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > For additional commands, e-mail: dev-help@maven.apache.org
> > >
> > >
> >
>