You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ant.apache.org by st...@apache.org on 2007/02/14 16:43:09 UTC
svn commit: r507587 - /ant/core/trunk/docs/manual/CoreTasks/parallel.html
Author: stevel
Date: Wed Feb 14 07:43:08 2007
New Revision: 507587
URL: http://svn.apache.org/viewvc?view=rev&rev=507587
Log:
thread-safe edition
Modified:
ant/core/trunk/docs/manual/CoreTasks/parallel.html
Modified: ant/core/trunk/docs/manual/CoreTasks/parallel.html
URL: http://svn.apache.org/viewvc/ant/core/trunk/docs/manual/CoreTasks/parallel.html?view=diff&rev=507587&r1=507586&r2=507587
==============================================================================
--- ant/core/trunk/docs/manual/CoreTasks/parallel.html (original)
+++ ant/core/trunk/docs/manual/CoreTasks/parallel.html Wed Feb 14 07:43:08 2007
@@ -26,8 +26,32 @@
<h2>Parallel</h2>
<h3>Description</h3>
-<p>Parallel is a container task - it can contain other Ant tasks. Each nested
-task within the parallel task will be executed in its own thread. </p>
+<p>
+ Executes nested tasks in parallel with no guarantees of thread safety.
+ Every task will run in its own thread, with the likelihood of
+ concurrency problems scaling with the number of CPUs on the host system.
+</p>
+<p><b>Warning:</b> While the Ant core is believed to be thread safe, no such
+ guarantees are made about tasks, which are not tested for thread safety during
+ Ant's test process.
+ Third party tasks may or may not be thread safe, and some of Ant's core tasks, such as
+ <code><javac></code> are definitely not re-entrant. This is because they use libraries that
+ were never designed to be used in a multithreaded environment.
+</p>
+<p>
+ The primary use case for <code><parallel></code> is to run external programs
+ such as an application server, and the JUnit or TestNG test suites at the
+ same time. Anyone trying to run large Ant task sequences in parallel, such
+ as javadoc and javac at the same time, is implicitly taking on the task
+ of identifying and fixing all concurrency bugs the tasks that they run.
+
+</p>
+<p>
+ Accordingly, while this task has uses, it should be considered an advanced
+ task which should be used in certain batch-processing or testing situations,
+ rather than an easy trick to speed up build times on a multiway CPU.
+</p>
+
<h3>Parameters</h3>
<table border="1" cellpadding="2" cellspacing="0">
<tr>
@@ -43,15 +67,10 @@
<tr>
<td valign="top">threadsPerProcessor</td>
<td valign="top">Maximum number of threads to use per available processor
-(Requires JDK 1.4)</td>
+(Java 1.4+)</td>
<td align="center" valign="top">No, defers to threadCount</td>
</tr>
<tr>
- <td valign="top">pollInterval</td>
- <td valign="top">Currently has no effect</td>
- <td align="center" valign="top">No, default is 1000</td>
- </tr>
- <tr>
<td valign="top">timeout</td>
<td valign="top">Number of milliseconds before execution is terminated</td>
<td align="center" valign="top">No</td>
@@ -62,28 +81,29 @@
at that point without waiting for any other tasks to complete.</td>
<td align="center" valign="top">No</td>
</tr>
+ <tr>
+ <td valign="top">pollInterval</td>
+ <td valign="top">Currently has no effect</td>
+ <td align="center" valign="top">No, default is 1000</td>
+ </tr>
</table>
<p>Parallel tasks have a number of uses in an Ant build file including:</p>
<ul>
-<li>Taking advantage of available processing resources to reduce build time</li>
+<li>Taking advantage of available processing resources to excute external
+ programs simultaneously.</li>
<li>Testing servers, where the server can be run in one thread and the test
harness is run in another thread.</li>
</ul>
-<p>Care must be taken when using multithreading to ensure the tasks within the
-threads do not interact. For example, two javac compile tasks which write
-classes into the same destination directory may interact where one tries to
-read a class for dependency information while the other task is writing the
-class file. Be sure to avoid these types of interactions within a
-<code><parallel></code> task</p>
-
<p>Any valid Ant task may be embedded within a
-parallel task, including other parallel tasks.</p>
+parallel task, including other parallel tasks, though there is no guarantee that
+the tasks will be thread safe in such an environment.</p>
-<p>Note that while the tasks within the parallel task are being run, the main
+<p>While the tasks within the parallel task are being run, the main
thread will be blocked waiting for all the child threads to complete. If
-execution is terminated by a timeout or a nested task failure when the failonany
+execution is terminated by a timeout or a nested task failure when the
+<code>failonany</code>
flag is set, the parallel task will complete without waiting for other nested
tasks to complete in other threads.
</p>
@@ -96,7 +116,7 @@
sequential</a> task to define sequences of tasks to be executed on each thread
within the parallel block</p>
-<p>The threadCount attribute can be used to place a maximum number of available
+<p>The <code>threadCount</code> attribute can be used to place a maximum number of available
threads for the execution. When not present all child tasks will be executed at
once. When present then the maximum number of concurrently executing tasks will
not exceed the number of threads specified. Furthermore, each task will be
@@ -104,16 +124,18 @@
of execution or the order of completion of the tasks, only that each will be
started before the next.<p>
-<p>If you are using J2RE 1.4 or later you can also use the threadsPerProcessor
+<p>If you are using Java 1.4 or later you can also use the <code>threadsPerProcessor</code>
and the number of available threads will be the stated multiple of the number of
processors (there is no affinity to a particular processor however). This will
-override the value in threadCount. If threadsPerProcessor is specified using
-any version prior to 1.4 then the value in threadCount will be used as is.</p>
+override the value in <code>threadCount</code>. If <code>threadsPerProcessor</code>
+is specified on any older JVM, then the value in <code>threadCount</code> will be used as is.</p>
-<p>When using threadCount and threadsPerProcessor care should be taken to ensure
-that the build does not deadlock. This can be caused by tasks such as waitFor
-taking up all available threads before the tasks that would unlock the waitfor
-would occur. This is not a repalcement for Java Language level thread
+<p>When using <code>threadCount</code> and <code>threadsPerProcessor</code>
+ care should be taken to ensure that the build does not deadlock.
+ This can be caused by tasks such as <code>waitfor</code>
+ taking up all available threads before the tasks that would unlock the
+ <code>waitfor</code>
+would occur. This is not a replacement for Java Language level thread
semantics and is best used for "embarassingly parallel" tasks.</p>
@@ -141,57 +163,71 @@
<wlrun ... >
<sequential>
<sleep seconds="30"/>
- <junit ... >
+ <junit fork="true" forkmode="once" ... >
<wlstop/>
</sequential>
</parallel>
</pre>
<p>This example represents a typical pattern for testing a server application.
-In one thread the server is started (the wlrun task). The other thread consists
-of a three tasks which are performed in sequence. The sleep task is used to
+In one thread the server is started (the <code><wlrun></code> task).
+The other thread consists
+of a three tasks which are performed in sequence. The <code><sleep></code> task is used to
give the server time to come up. Another task which is capable of validating
-that the server is available could be used in place of the sleep task. The
-test harness is then run. Once the tests are complete, the server is stopped
-(using wlstop in this example), allowing both threads to complete. The
-parallel task will also complete at this time and the build will then
+that the server is available could be used in place of the <code><sleep></code> task. The
+<code><junit></code> test harness then runs, again in its own JVM. Once the tests are complete, the server is stopped
+(using <code><wlstop></code> in this example), allowing both threads to complete. The
+<code><parallel></code> task will also complete at this time and the build will then
continue.</p>
<pre>
<parallel>
- <javac ...> <!-- compiler servlet code -->
+ <javac fork="true"...> <!-- compiler servlet code -->
<wljspc ...> <!-- precompile JSPs -->
</parallel>
</pre>
<p>This example shows two independent tasks being run to achieve better
resource utilization during the build. In this instance, some servlets are being
-compiled in one thead and a set of JSPs is being precompiled in another. As
-noted above, you need to be careful that the two tasks are independent, both in
+compiled in one thead and a set of JSPs is being precompiled in another. Developers
+need to be careful that the two tasks are independent, both in
terms of their dependencies and in terms of their potential interactions in
-Ant's external environment.</p>
+Ant's external environment. Here we set <code>fork="true"</code> for the
+<code><javac></code> task, so that it runs in a new process;
+if the <code><wljspc></code> task used the javac compiler in-VM
+(it may), concurrency problems may arise.
+</p>
+<taskdef name=""
<pre>
+ <macrodef name="dbpurge">
+ <attribute file="file"/>
+ <sequential>
+ <java jar="utils/dbpurge.jar" fork="true" >
+ <arg file="@{file} />
+ </java>
+ </sequential>
+</macrodef>
+
<parallel threadCount='4'>
- <ant target='TargetThatConsumesLotsOfCPUTimeAndMemory'>
- <param name='file' value='one.txt'/>
- </ant>
- <ant target='TargetThatConsumesLotsOfCPUTimeAndMemory'>
- <param name='file' value='two.txt'/>
- </ant>
- <ant target='TargetThatConsumesLotsOfCPUTimeAndMemory'>
- <param name='file' value='three.txt'/>
- </ant>
+ <dbpurge file="db/one" />
+ <dbpurge file="db/two" />
+ <dbpurge file="db/three" />
+ <dbpurge file="db/four" />
+ <dbpurge file="db/five" />
+ <dbpurge file="db/six" />
+ <dbpurge file="db/seven" />
+ <dbpurge file="db/eight" />
<!-- repeated about 40 times -->
</parallel>
</pre>
<p>This example represents a typical need for use of the threadCount and
threadsPerProcessor attributes. Spinning up all 40 of those tasks could cripple
-the JVM for memory and the CPU for available time. By limiting the number of
-concurrent executions you can get the task done in about the same assuming
-infinite memory time without needing infinite memory. This is also a good
+the system for memory and CPU time. By limiting the number of
+concurrent executions you can reduce contention for CPU, memory and disk IO,
+and so actually finish faster. This is also a good
candidiate for use of threadCount (and possibly threadsPerProcessor) because
-each task (in this hypothetical case) is independent and has no dependencies on
+each task is independent (every new JVM is forked) and has no dependencies on
the other tasks.</p>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org
Re: svn commit: r507587 - /ant/core/trunk/docs/manual/CoreTasks/parallel.html
Posted by Steve Loughran <st...@apache.org>.
stevel@apache.org wrote:
> Author: stevel
> Date: Wed Feb 14 07:43:08 2007
> New Revision: 507587
>
> URL: http://svn.apache.org/viewvc?view=rev&rev=507587
> Log:
> thread-safe edition
I decided to make it very clear that Ant makes no thread safety
guarantees about its classes, and moved all examples to use forking
wherever appropriate, to emphasise thsi fact.
> +<p>
> + The primary use case for <code><parallel></code> is to run external programs
> + such as an application server, and the JUnit or TestNG test suites at the
> + same time. Anyone trying to run large Ant task sequences in parallel, such
> + as javadoc and javac at the same time, is implicitly taking on the task
> + of identifying and fixing all concurrency bugs the tasks that they run.
> +
> +</p>
> +<p>
> + Accordingly, while this task has uses, it should be considered an advanced
> + task which should be used in certain batch-processing or testing situations,
> + rather than an easy trick to speed up build times on a multiway CPU.
> +</p>
Seem reasonable. I dont want to put people off using it; (FWIW its the
basis of our <functionaltest> task that I would love to move from
smartfrog into ant), but I do want anyone who picks up the task to
recognise the responsibilities.
-Steve
(who has to deal with RMI-reentrancy across clusters of machines)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org