You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ant.apache.org by Antoine Reilles <An...@loria.fr> on 2006/05/29 17:48:48 UTC
subant and memory problems
Hi,
For building the examples of the tom project (http://tom.loria.fr), we
use the subant task, each example containing a build.xml snippet, using
<target name="build.all">
<subant target="build">
<fileset dir="." includes="*/build.xml"/>
</subant>
</target>
Those examples do use the tom ant task to compile tom sources to java,
and then javac. The tom ant task require to load the tom compiler, which
is a java application. However, we currently notice memory problems when
building all examples. For each example, ant do load all tom's compiler
classes, but it seems they are never released. thus, after a few
examples, ant dies with an OutOfMemory error. Buildinf with -v -d shows
ant do load the classes of the tom compiler and all dependancies for
each subant task, with
[snip]
Finding class aterm.ATermAppl
Loaded from /home/tonio/workspace/jtom/src/dist/lib/aterm.jar
aterm/ATermAppl.class
Class aterm.ATermAppl loaded from ant loader (parentFirst)
[snip]
However, there is no mentions of unloading.
Is it possible that ant do retain some memory, and thus causes the
failure ? Or are we using the subant task incorrectly ?
Best regards,
antoine
Re: subant and memory problems
Posted by Antoine Reilles <An...@loria.fr>.
On Mon, Jun 05, 2006 at 10:44:27AM -0500, Dominique Devienne wrote:
> >Ok, great ! I just saw the code to the jaxb task. Indeed, it looks quite
> >easy to use the Java task for that. Thanks a lot for the pointer !
>
> Always glad to help ;-)
Wow, i tried using Java in the tom task to run the compiler:
With this new task, there is no more memory problem !
I imagine the neste Java take care of classloading and stuff.
My old task was simply calling a static function in the tom compiler,
and was then causing the load of all tom compiler classes at the ant
task level at each run of the task, while the nested Java is not.
Re: subant and memory problems
Posted by Dominique Devienne <dd...@gmail.com>.
> Ok, great ! I just saw the code to the jaxb task. Indeed, it looks quite
> easy to use the Java task for that. Thanks a lot for the pointer !
Always glad to help ;-)
> I did that, running tom compilation from a java app for fifteen files in
> an infinite loop: it ran for 2 days without showing any memory hungry
> behavior. There still may be some static containers that would need
> cleanup, but even then, i don't think they could eat that much memory,
> and don't show in my testcase.
Maybe this is just showing good caching ;-) OK, ok, maybe not that
funny. But in general, it can be tricky to devise a test case that
simulate real world usage, so maybe compiling the same 15 files again
and again isn't triggering the same code the same way?
Note that I'm not saying Ant is leak free. Most recent leaks have been
fixed by non-regular contributors bugged enough by the leaks. We just
hope you've reached that point that you'd fix Ant for us! ;-) --DD
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: subant and memory problems
Posted by Antoine Reilles <An...@loria.fr>.
Hi,
On Mon, Jun 05, 2006 at 09:05:03AM -0500, Dominique Devienne wrote:
> >I guess the more elegant way would be to add a "fork" attribute to our
> >tom task, but i don't know how difficult it would be, as to is simply a
> >java application.
>
> That would be the way to go, indeed. It's rather simple to compose the
> Java task in another task. I've used this approach to add "smarts" to
> the Jaxb compiler for example. The code is even in bugzilla I think,
> if you want to have a look. The <tom> task would do all Ant related
> processing, check timestamps and all, and simply generate the
> (possibly long) command line to pass to Java (or Execute) to invoke
> Tom in another VM. --DD
Ok, great ! I just saw the code to the jaxb task. Indeed, it looks quite
easy to use the Java task for that. Thanks a lot for the pointer !
>
> PS: To investigate whether Tom suffers from the ill as Javac used to,
> you may want to write a pure Java program (no Ant involved) that
> invokes Tom several time in a loop, and see whether memory usage
> increases because memory is not released. Also, check the Tom task
> resets refs to Tom data structures or engine to null. Check Tom
> classes don't have static caches. These kind of things.
I did that, running tom compilation from a java app for fifteen files in
an infinite loop: it ran for 2 days without showing any memory hungry
behavior. There still may be some static containers that would need
cleanup, but even then, i don't think they could eat that much memory,
and don't show in my testcase.
Cheers,
antoine
Re: subant and memory problems
Posted by Dominique Devienne <dd...@gmail.com>.
> I guess the more elegant way would be to add a "fork" attribute to our
> tom task, but i don't know how difficult it would be, as to is simply a
> java application.
That would be the way to go, indeed. It's rather simple to compose the
Java task in another task. I've used this approach to add "smarts" to
the Jaxb compiler for example. The code is even in bugzilla I think,
if you want to have a look. The <tom> task would do all Ant related
processing, check timestamps and all, and simply generate the
(possibly long) command line to pass to Java (or Execute) to invoke
Tom in another VM. --DD
PS: To investigate whether Tom suffers from the ill as Javac used to,
you may want to write a pure Java program (no Ant involved) that
invokes Tom several time in a loop, and see whether memory usage
increases because memory is not released. Also, check the Tom task
resets refs to Tom data structures or engine to null. Check Tom
classes don't have static caches. These kind of things.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: subant and memory problems
Posted by Antoine Reilles <An...@loria.fr>.
On Sun, Jun 04, 2006 at 09:17:13PM +0100, Steve Loughran wrote:
> Antoine Reilles wrote:
> >Hi,
> >
> >For building the examples of the tom project (http://tom.loria.fr), we
> >use the subant task, each example containing a build.xml snippet, using
> > <target name="build.all">
> > <subant target="build">
> > <fileset dir="." includes="*/build.xml"/>
> > </subant>
> > </target>
> >Those examples do use the tom ant task to compile tom sources to java,
> >and then javac. The tom ant task require to load the tom compiler, which
> >is a java application. However, we currently notice memory problems when
> >building all examples. For each example, ant do load all tom's compiler
> >classes, but it seems they are never released. thus, after a few
> >examples, ant dies with an OutOfMemory error. Buildinf with -v -d shows
> >ant do load the classes of the tom compiler and all dependancies for
> >each subant task, with
> >
> >[snip]
> >Finding class aterm.ATermAppl
> >Loaded from /home/tonio/workspace/jtom/src/dist/lib/aterm.jar
> >aterm/ATermAppl.class
> >Class aterm.ATermAppl loaded from ant loader (parentFirst)
> >[snip]
> >
> >However, there is no mentions of unloading.
> >Is it possible that ant do retain some memory, and thus causes the
> >failure ? Or are we using the subant task incorrectly ?
> Is there any way to run the compiler in a new process?
I tries the <apply> approach Dominique suggested. It works fine, but i
had to translate the simple
<subant target="build">
<fileset dir="." includes="*/build.xml"/>
</subant>
by something much less elegant
<apply executable="ant"
parallel="false"
failonerror="yes">
<arg value="-Dtom.home=${tom.home}"/>
<arg value="-f"/>
<fileset dir="." includes="*/build.xml"/>
</apply>
where i need to pass the interesting properties explicitly
This solution is also quite cpu/io expensive
> some compilers (javac was always one) used to build up massive
> datastructures that would never get released...
Tom also build some massive datastructures and abstract syntax trees,
but those ast are java implementation generated by the gom
tool, and are not faulty here.
I guess the more elegant way would be to add a "fork" attribute to our
tom task, but i don't know how difficult it would be, as to is simply a
java application. We could also simply run the tom compiler using the <java>
task with fork=yes, but that would lead to quite complex macrodef
Cheers,
antoine
Re: subant and memory problems
Posted by Steve Loughran <st...@apache.org>.
Antoine Reilles wrote:
> Hi,
>
> For building the examples of the tom project (http://tom.loria.fr), we
> use the subant task, each example containing a build.xml snippet, using
> <target name="build.all">
> <subant target="build">
> <fileset dir="." includes="*/build.xml"/>
> </subant>
> </target>
> Those examples do use the tom ant task to compile tom sources to java,
> and then javac. The tom ant task require to load the tom compiler, which
> is a java application. However, we currently notice memory problems when
> building all examples. For each example, ant do load all tom's compiler
> classes, but it seems they are never released. thus, after a few
> examples, ant dies with an OutOfMemory error. Buildinf with -v -d shows
> ant do load the classes of the tom compiler and all dependancies for
> each subant task, with
>
> [snip]
> Finding class aterm.ATermAppl
> Loaded from /home/tonio/workspace/jtom/src/dist/lib/aterm.jar
> aterm/ATermAppl.class
> Class aterm.ATermAppl loaded from ant loader (parentFirst)
> [snip]
>
> However, there is no mentions of unloading.
> Is it possible that ant do retain some memory, and thus causes the
> failure ? Or are we using the subant task incorrectly ?
>
Is there any way to run the compiler in a new process?
some compilers (javac was always one) used to build up massive
datastructures that would never get released...
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: subant and memory problems
Posted by Dominique Devienne <dd...@gmail.com>.
On 5/31/06, Antoine Reilles <An...@loria.fr> wrote:
> Also, if I change the build to avoid the use of subant, by inlining all
> subprocesses, the build goes farther, since it loads all tom's compiler
> classes only once, instead of once for each subant part.
Using <subant>, just like <ant>, creates a new project, which
duplicates the property and reference maps, and probably other stuff
too. It's not surprising it's taking more memory than using a single
project. Ant (and SubAnt, which just re-uses Ant) was an afterthought
it looks to me, and is thus very inefficient memory-wise.
OTOH, I don't think the classes are reloaded. I don't see why the new
project would need to create a new class loader and reload the
classes.
I'm not sure whether the sub-project shares its ComponentHelper with
its parent project.
> We are still looking for memory problems in tom, but it looks like tom
> is not the only one at fault there.
Probably not. Ideally there would be a fork attribute for <ant> (and
<subant>) to ensure each sub-build runs in its own VM, ensuring proper
release of memory (when the process exists), but it's never been done
and is non-trivial, since a cross-process logger/listener thing needs
to be put in place, and what to do with references? Serialize them to
the sub-project? Not much is serializable in Ant.
To solve your problem, you may have to switch to <apply>, which forks
a new Ant JVM for each sub-build (in non-parallel mode), which would
emulate fork=true for <ant>. --DD
PS: The fact that it still OOME even with inline compiles kinda
indicates the memory used by Tom for an earlier compile is not
released when memory is getting scarce, so strong ref. to big data is
still being help on to. That's the same symptom as Javac used to
exhibit before 1.4.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org
Re: subant and memory problems
Posted by Antoine Reilles <An...@loria.fr>.
On Tue, May 30, 2006 at 07:49:26AM -0500, Dominique Devienne wrote:
> It's quite possible Ant or SubAnt itself hold on to too many
> references. There have been fixes in Ant's ComponentHelper in the
> past. But maybe Tom's compiler could be at fault as well, no? This
> would need to be investigated using a JProbe-like tool.
I tried to trace a build run under jprofiler, but it didn't show obvious
problems in tom.
Also, if I change the build to avoid the use of subant, by inlining all
subprocesses, the build goes farther, since it loads all tom's compiler
classes only once, instead of once for each subant part.
We are still looking for memory problems in tom, but it looks like tom
is not the only one at fault there.
Cheers,
antoine
Re: subant and memory problems
Posted by Dominique Devienne <dd...@gmail.com>.
It's quite possible Ant or SubAnt itself hold on to too many
references. There have been fixes in Ant's ComponentHelper in the
past. But maybe Tom's compiler could be at fault as well, no? This
would need to be investigated using a JProbe-like tool.
--DD
On 5/29/06, Antoine Reilles <An...@loria.fr> wrote:
> Hi,
>
> For building the examples of the tom project (http://tom.loria.fr), we
> use the subant task, each example containing a build.xml snippet, using
> <target name="build.all">
> <subant target="build">
> <fileset dir="." includes="*/build.xml"/>
> </subant>
> </target>
> Those examples do use the tom ant task to compile tom sources to java,
> and then javac. The tom ant task require to load the tom compiler, which
> is a java application. However, we currently notice memory problems when
> building all examples. For each example, ant do load all tom's compiler
> classes, but it seems they are never released. thus, after a few
> examples, ant dies with an OutOfMemory error. Buildinf with -v -d shows
> ant do load the classes of the tom compiler and all dependancies for
> each subant task, with
>
> [snip]
> Finding class aterm.ATermAppl
> Loaded from /home/tonio/workspace/jtom/src/dist/lib/aterm.jar
> aterm/ATermAppl.class
> Class aterm.ATermAppl loaded from ant loader (parentFirst)
> [snip]
>
> However, there is no mentions of unloading.
> Is it possible that ant do retain some memory, and thus causes the
> failure ? Or are we using the subant task incorrectly ?
>
> Best regards,
> antoine
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
For additional commands, e-mail: user-help@ant.apache.org