You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Jörg Schaible <jo...@swisspost.com> on 2014/10/16 11:12:49 UTC
Regression: Reactor memory consumption/leak
Hi folks,
we have a single build with currently ~400 projects (incl. builders i.e.
POMs having modules only). We are already used to increase the provided
memory in MAVEN_OPTS, but lately we have troubles to build at all because of
OOMEs (heap). Look at following numbers building with the different Maven
versions (latest Oracle JDK 7 on 64-bit Linux):
====================== %< =======================
$ MAVEN_OPTS="-Xmx640m" mvn-3.0.5 validate
...
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 25.392s
[INFO] Finished at: Thu Oct 16 10:02:56 CEST 2014
[INFO] Final Memory: 276M/511M
[INFO] -----------------------------------------------------------------
====================== %< =======================
3.0.5 fails with 512 only.
====================== %< =======================
MAVEN_OPTS="-Xmx1100m" mvn-3.1.1 validate
...
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 37.004s
[INFO] Finished at: Thu Oct 16 10:13:36 CEST 2014
[INFO] Final Memory: 431M/978M
[INFO] -----------------------------------------------------------------
====================== %< =======================
3.1.1 fails with 1024m only.
====================== %< =======================
MAVEN_OPTS="-Xmx2500m" mvn-3.2.3 validate
...
[INFO] -----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 55.465 s
[INFO] Finished at: 2014-10-16T10:27:42+02:00
[INFO] Final Memory: 1488M/2222M
[INFO] -----------------------------------------------------------------
====================== %< =======================
3.2.3 fails with 2400m only.
The required memory increases dramatically from Maven version to version
just to calculate the build order list (the OOME occurs before this list is
printed).
- Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Kristian Rosenvold <kr...@gmail.com>.
You could take some heap dumps and share them with me on google disk.
A baseline from the best one and one from the worst one. It would be
nice if you could get them at approx the same time in the build
Kristian
2014-10-16 15:20 GMT+02:00 Jörg Schaible <jo...@swisspost.com>:
> Hi Igor,
>
> Igor Fedorenko wrote:
>
>> Can you provide an example project we can use to reproduce the problem
>> locally? You may be able to strip your source tree from everything bun
>> pom.xml files, for example.
>
> Interesting idea. I'll try if this works out.
>
> - Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Jörg Schaible <jo...@swisspost.com>.
Hi Igor,
Igor Fedorenko wrote:
> Can you provide an example project we can use to reproduce the problem
> locally? You may be able to strip your source tree from everything bun
> pom.xml files, for example.
Interesting idea. I'll try if this works out.
- Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Mark Derricutt <ma...@talios.com>.
On 18 Oct 2014, at 10:10, Jörg Schaible wrote:
> thanks again for taking the time. Something must have gone wrong for an
> increased memory usage by 400% for the same stuff.
Interesting - this might explain why a lot of my builds are now blowing up our build server ever since we started using our new tiles-maven-plugin in bulk ( the worst culprit being our last remaining multi-module build ).
Since we're now dynamically adding 3-4 parents into build chain, this issue might be manifesting even more ( tho I suspect one of newer plugins I added to one of our tiles also has a leak ).
Re: Regression: Reactor memory consumption/leak
Posted by Jörg Schaible <jo...@gmx.de>.
Hi Igor,
Igor Fedorenko wrote:
> I think I know the problem.
>
> Current master creates multiple MavenProject instances for the same
> parent pom, while 3.0.5 reused the same instance. I have not looked how
> hard this will be to fix yet.
>
> In, more generally, MavenProject implementation is far from efficient.
> For example, each instance holds two copies of project Model and
> absolutely no model element reuse. For example, the same <dependency>
> element defined in a parent pom will be represented as distinct object
> instance in each child project. This is something we've known in m2e for
> long time, but optimizing this will quite tricky especially if we want
> to maintain backwards compatibility and keep model instances mutable.
thanks again for taking the time. Something must have gone wrong for an
increased memory usage by 400% for the same stuff.
Cheers,
Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
I believe there are several issues
I don't know what caused memory usage increase in 3.1.1. Memory increase
in 3.2.3 is likely related to MNG-5669 [1]. Then there is general
inefficiency in how Maven represents project Model in memory.
The memory increase in 3.2.3 may be possible to workaround, if you
really have to. The problem is triggered by <parent/> elements that
point at the same pom.xml file resolved from maven repositories.
Intermediate parent pom part of the same reactor build should mask the
problem. I believe this is how most projects are setup, which explains
why the problem affects relativly small number of users.
[1] http://jira.codehaus.org/browse/MNG-5669
--
Regards,
Igor
On 2014-10-23, 1:27, Milos Kleint wrote:
> is there an issue to watch?
>
> Thanks
>
> Milos
>
> On Fri, Oct 17, 2014 at 10:59 PM, Igor Fedorenko <ig...@ifedorenko.com>
> wrote:
>
>> I think I know the problem.
>>
>> Current master creates multiple MavenProject instances for the same
>> parent pom, while 3.0.5 reused the same instance. I have not looked how
>> hard this will be to fix yet.
>>
>> In, more generally, MavenProject implementation is far from efficient.
>> For example, each instance holds two copies of project Model and
>> absolutely no model element reuse. For example, the same <dependency>
>> element defined in a parent pom will be represented as distinct object
>> instance in each child project. This is something we've known in m2e for
>> long time, but optimizing this will quite tricky especially if we want
>> to maintain backwards compatibility and keep model instances mutable.
>>
>> --
>> Regards,
>> Igor
>>
>>
>>
>> On 2014-10-16, 13:30, Jörg Schaible wrote:
>>
>>> Hi Igor,
>>>
>>> Igor Fedorenko wrote:
>>>
>>> You can zip and email it to me directly or share it on github, dropbox
>>>> or google drive and send me the link. I am flexible :-)
>>>>
>>>
>>> I've sent the small one directly ...
>>>
>>> Cheers,
>>> Jörg
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>> For additional commands, e-mail: dev-help@maven.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Milos Kleint <mk...@gmail.com>.
is there an issue to watch?
Thanks
Milos
On Fri, Oct 17, 2014 at 10:59 PM, Igor Fedorenko <ig...@ifedorenko.com>
wrote:
> I think I know the problem.
>
> Current master creates multiple MavenProject instances for the same
> parent pom, while 3.0.5 reused the same instance. I have not looked how
> hard this will be to fix yet.
>
> In, more generally, MavenProject implementation is far from efficient.
> For example, each instance holds two copies of project Model and
> absolutely no model element reuse. For example, the same <dependency>
> element defined in a parent pom will be represented as distinct object
> instance in each child project. This is something we've known in m2e for
> long time, but optimizing this will quite tricky especially if we want
> to maintain backwards compatibility and keep model instances mutable.
>
> --
> Regards,
> Igor
>
>
>
> On 2014-10-16, 13:30, Jörg Schaible wrote:
>
>> Hi Igor,
>>
>> Igor Fedorenko wrote:
>>
>> You can zip and email it to me directly or share it on github, dropbox
>>> or google drive and send me the link. I am flexible :-)
>>>
>>
>> I've sent the small one directly ...
>>
>> Cheers,
>> Jörg
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>> For additional commands, e-mail: dev-help@maven.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
Re: Regression: Reactor memory consumption/leak
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
I think I know the problem.
Current master creates multiple MavenProject instances for the same
parent pom, while 3.0.5 reused the same instance. I have not looked how
hard this will be to fix yet.
In, more generally, MavenProject implementation is far from efficient.
For example, each instance holds two copies of project Model and
absolutely no model element reuse. For example, the same <dependency>
element defined in a parent pom will be represented as distinct object
instance in each child project. This is something we've known in m2e for
long time, but optimizing this will quite tricky especially if we want
to maintain backwards compatibility and keep model instances mutable.
--
Regards,
Igor
On 2014-10-16, 13:30, Jörg Schaible wrote:
> Hi Igor,
>
> Igor Fedorenko wrote:
>
>> You can zip and email it to me directly or share it on github, dropbox
>> or google drive and send me the link. I am flexible :-)
>
> I've sent the small one directly ...
>
> Cheers,
> Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Jörg Schaible <jo...@swisspost.com>.
Hi Igor,
Igor Fedorenko wrote:
> You can zip and email it to me directly or share it on github, dropbox
> or google drive and send me the link. I am flexible :-)
I've sent the small one directly ...
Cheers,
Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
You can zip and email it to me directly or share it on github, dropbox
or google drive and send me the link. I am flexible :-)
--
Regards,
Igor
On 2014-10-16, 11:41, Jörg Schaible wrote:
> Hi Igor,
>
> Igor Fedorenko wrote:
>
>> Can you provide an example project we can use to reproduce the problem
>> locally? You may be able to strip your source tree from everything bun
>> pom.xml files, for example.
>
> OK, this works out. I have now such a transportable setup. It's zipped about
> 1.1MB with an empty local repository or 38MB with a filled one. I can run
>
> MAVEN_OPTS="-Xmx3000m" mvn validate -s settings.xml
>
> successfully and will fail with OOME calling
>
> MAVEN_OPTS="-Xmx2000m" mvn validate -s settings.xml
>
> using M323/Java7 on Linux x64. The project tree contains the pom.xml and
> anything that triggers profiles automatically. Any non-public artifact is in
> a "remote" repo (included in the 1.1MB), so you should be able to reproduce
> this. Which do you want and what's the most useful place to put this?
>
> Cheers,
> Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Jörg Schaible <jo...@swisspost.com>.
Hi Igor,
Igor Fedorenko wrote:
> Can you provide an example project we can use to reproduce the problem
> locally? You may be able to strip your source tree from everything bun
> pom.xml files, for example.
OK, this works out. I have now such a transportable setup. It's zipped about
1.1MB with an empty local repository or 38MB with a filled one. I can run
MAVEN_OPTS="-Xmx3000m" mvn validate -s settings.xml
successfully and will fail with OOME calling
MAVEN_OPTS="-Xmx2000m" mvn validate -s settings.xml
using M323/Java7 on Linux x64. The project tree contains the pom.xml and
anything that triggers profiles automatically. Any non-public artifact is in
a "remote" repo (included in the 1.1MB), so you should be able to reproduce
this. Which do you want and what's the most useful place to put this?
Cheers,
Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Igor Fedorenko <ig...@ifedorenko.com>.
Can you provide an example project we can use to reproduce the problem
locally? You may be able to strip your source tree from everything bun
pom.xml files, for example.
--
Regards,
Igor
On 2014-10-16, 5:12, Jörg Schaible wrote:
> Hi folks,
>
> we have a single build with currently ~400 projects (incl. builders i.e.
> POMs having modules only). We are already used to increase the provided
> memory in MAVEN_OPTS, but lately we have troubles to build at all because of
> OOMEs (heap). Look at following numbers building with the different Maven
> versions (latest Oracle JDK 7 on 64-bit Linux):
>
> ====================== %< =======================
> $ MAVEN_OPTS="-Xmx640m" mvn-3.0.5 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 25.392s
> [INFO] Finished at: Thu Oct 16 10:02:56 CEST 2014
> [INFO] Final Memory: 276M/511M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.0.5 fails with 512 only.
>
>
> ====================== %< =======================
> MAVEN_OPTS="-Xmx1100m" mvn-3.1.1 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 37.004s
> [INFO] Finished at: Thu Oct 16 10:13:36 CEST 2014
> [INFO] Final Memory: 431M/978M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.1.1 fails with 1024m only.
>
>
> ====================== %< =======================
> MAVEN_OPTS="-Xmx2500m" mvn-3.2.3 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 55.465 s
> [INFO] Finished at: 2014-10-16T10:27:42+02:00
> [INFO] Final Memory: 1488M/2222M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.2.3 fails with 2400m only.
>
>
> The required memory increases dramatically from Maven version to version
> just to calculate the build order list (the OOME occurs before this list is
> printed).
>
> - Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Jörg Schaible <jo...@swisspost.com>.
Hi Jason,
Jason van Zyl wrote:
> Igor and I have been moving a vast project with hundreds of modules (200
> to 400 to 500) from 3.1.0 through 3.2.4-SNAPSHOT and don't observe this
> drastic change. We would definitely notice.
>
> Are all your plugin versions locked down in that they don't vary even
> though the version of Maven does?
Except eclipse and help plugin - yes.
> Also are you running reporting in here?
In validate? No. However, it does not reach this point at all:
====================== %< ======
$ mvn-3.2.3 validate
[INFO] Scanning for projects...
Downloading: http://es3.elsag.de/~maven/repo-m2-snapshot/com/scalaris/buildsystem/maven2/master/x-SNAPSHOT/maven-metadata.xml
Downloaded: http://es3.elsag.de/~maven/repo-m2-snapshot/com/scalaris/buildsystem/maven2/master/x-SNAPSHOT/maven-metadata.xml (813 B at 10.7 KB/sec)
Downloading: http://es3.elsag.de/~maven/repo-m2-snapshot/ip/ip-master/5.0.x-SNAPSHOT/maven-metadata.xml
Downloading: http://es3.elsag.de/~maven/repo-m2-snapshot/ip/ip-master/4.4.x-SNAPSHOT/maven-metadata.xml
Downloading: http://es3.elsag.de/~maven/repo-m2-snapshot/com/essencio/components/mobile/master/5.0.x-SNAPSHOT/maven-metadata.xml
[ERROR] GC overhead limit exceeded -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError
====================== %< =======================
- Jörg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Regression: Reactor memory consumption/leak
Posted by Jason van Zyl <ja...@takari.io>.
Igor and I have been moving a vast project with hundreds of modules (200 to 400 to 500) from 3.1.0 through 3.2.4-SNAPSHOT and don't observe this drastic change. We would definitely notice.
Are all your plugin versions locked down in that they don't vary even though the version of Maven does? Also are you running reporting in here?
On Oct 16, 2014, at 5:12 AM, Jörg Schaible <jo...@swisspost.com> wrote:
> Hi folks,
>
> we have a single build with currently ~400 projects (incl. builders i.e.
> POMs having modules only). We are already used to increase the provided
> memory in MAVEN_OPTS, but lately we have troubles to build at all because of
> OOMEs (heap). Look at following numbers building with the different Maven
> versions (latest Oracle JDK 7 on 64-bit Linux):
>
> ====================== %< =======================
> $ MAVEN_OPTS="-Xmx640m" mvn-3.0.5 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 25.392s
> [INFO] Finished at: Thu Oct 16 10:02:56 CEST 2014
> [INFO] Final Memory: 276M/511M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.0.5 fails with 512 only.
>
>
> ====================== %< =======================
> MAVEN_OPTS="-Xmx1100m" mvn-3.1.1 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 37.004s
> [INFO] Finished at: Thu Oct 16 10:13:36 CEST 2014
> [INFO] Final Memory: 431M/978M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.1.1 fails with 1024m only.
>
>
> ====================== %< =======================
> MAVEN_OPTS="-Xmx2500m" mvn-3.2.3 validate
> ...
> [INFO] -----------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] -----------------------------------------------------------------
> [INFO] Total time: 55.465 s
> [INFO] Finished at: 2014-10-16T10:27:42+02:00
> [INFO] Final Memory: 1488M/2222M
> [INFO] -----------------------------------------------------------------
> ====================== %< =======================
> 3.2.3 fails with 2400m only.
>
>
> The required memory increases dramatically from Maven version to version
> just to calculate the build order list (the OOME occurs before this list is
> printed).
>
> - Jörg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
---------------------------------------------------------
There's no sense in being precise when you don't even know what you're talking about.
-- John von Neumann