[PATCH] multithreading tasks

[Conor MacNeill <co...@m64.com>]

Team,

Attached is a fairly simple patch plus a new class to allow ant to run tasks
in separate threads. The motivation for this patch is to allow us to perform
unit testing in a relatively automated fashion. In our particular situation
we would like to have an ant target for testing which will start weblogic in
one thread and JUnit in another. Once JUnit completes its tests, weblogic
will be stopped and the target will complete by joining all outstanding
threads.

This is not really motivated by trying to parallelize builds for performance
reasons but that may be possible too.

Whilst our particular use is for running weblogic, the changes here are
general in nature. I will post the taskdef for running weblogic in a later
post.

To get a task to run in a separate thread, simply add async="yes" to the
task definition.

Let me know what you think. If this is accepted, I plan two new small
taskdefs:- a sleep task and a join task.

Conor

Re: [PATCH] multithreading tasks

[Steve Loughran <st...@iseran.com>]

mmm. The Javac task itself might be an interesting experiment -just to see
if you get any speedup from parallelising javac tasks even on a single CPU
box -and what speedup you could get on a multiway system.

The complexity comes not from spawning off compliation into multiple
threads, but deciding what the main thread should do. In the existing ant
framework it would have to wait for the build to finish, so it would need to
either get on with its own compilation or just block till the multiple child
threads finished.

As to how much speedup you will get -who knows. When doing VC++ work on a
two way P6, I used to get up to 130% CPU utilisation -where 200% meant 'both
cores busy'. But C++ compliation may be more disk intensive (intermediate
files, linking, debug tables); you would have to try and see. In native code
I always focus on cache management ahead of multiway performance, except for
some very special cases; nothing beats a well placed prefetch.

    -Steve

"There is no meeting"

----- Original Message -----
From: "Bengt Bäverman" <be...@spray.se>
To: <an...@jakarta.apache.org>
Sent: Monday, June 19, 2000 11:27
Subject: Re: [PATCH] multithreading tasks


Is this not something that can be done in a normal taskdef? I don't see this
as necessary to include in the Ant application.

Bengt Bäverman
bengt.baverman@spray.se

Re: [PATCH] multithreading tasks

[Bengt Bäverman <be...@spray.se>]

Is this not something that can be done in a normal taskdef? I don't see this
as necessary to include in the Ant application.

Bengt Bäverman
bengt.baverman@spray.se

----- Original Message ----- 
From: "James Duncan Davidson" <ja...@eng.sun.com>
To: <an...@jakarta.apache.org>
Sent: Sunday, June 18, 2000 10:14 AM
Subject: Re: [PATCH] multithreading tasks


> on 5/28/00 8:58 AM, Conor MacNeill at conor@m64.com wrote:
> 
> > Attached is a fairly simple patch plus a new class to allow ant to run tasks
> > in separate threads. The motivation for this patch is to allow us to perform
> > unit testing in a relatively automated fashion. In our particular situation
> > we would like to have an ant target for testing which will start weblogic in
> > one thread and JUnit in another. Once JUnit completes its tests, weblogic
> > will be stopped and the target will complete by joining all outstanding
> > threads.
> 
> Has this been committed? I didn't see a commit message go out on it. In any
> case I'm -1 on this.
> 
> If concurency is needed for testing, then doing a test that that performs
> it's own parallelism at that level is fine, but building parallelism into
> Ant gets probelmatic fast. MT programming isn't *that* easy and I don't
> really think that projects should deal with it.
> 
> > Let me know what you think. If this is accepted, I plan two new small
> > taskdefs:- a sleep task and a join task.
> 
> Problem here is that now you are using tasks not to perform functionality,
> but as logical control. Sam has worked on me enough that I'm no longer
> totally freaked out about layering scripting into Ant, as long as that
> scripting isn't expressed in the XML. Expressing threading semantics in the
> XML is neat, but complicates things to a great degree.
> 
> I understand the motive -- parallelism on tests is a good thing, but I don't
> see why that functionality has to be genericized to this level.
> 
> .duncan
> 
> 

Re: [PATCH] multithreading tasks

[Thomas Haas <th...@softwired-inc.com>]

From: Conor MacNeill <co...@m64.com>
[deleted]
>
>
> OK, I've made my arguments. Perhaps you and others can comment?
>
> Cheers
>
> Conor
>


I have missed your patch and have of the discussion, but this is exactly
what we needind and building for the exact same purpose: running unit tests
against a server which needs to be started and stopped.
I am very interested in your code and in developing something together.

NOTE: I beleive general MT features are very difficult to add to ant and may
cause more problems than it solves, but this is NOT about general MT
solution, simply a task (or collection of tasks) needing another one to talk
to.

- tom

Re: [PATCH] multithreading tasks

[James Duncan Davidson <ja...@eng.sun.com>]

on 2000/06/22 04:22, Conor MacNeill at conor@cortexebusiness.com.au wrote:

Thanks for the XML sample. This helps a lot!

> <target name="test" depends="install, testcode">
> <wlrun taskname="server" async="yes"
> classpath="${weblogic.boot.classpath}"
> wlclasspath="${weblogic.classes}:..."
> name="${server.name}" home="${weblogic.home}"
> properties="${server.name}/${server.name}.properties"/>

Ok.. So I don't have any problem with the async flag being an attribute on a
task. Seems about right.

> <sleep delay="80000"/>

Yeah, sure it can be a task. But if this is telling the acutal Ant runtime
to pause for a while, how about a PI?

<?pause interval="880000"?>

Don't know if it's any better though. Especially since the async is an XML
attrib. Besides, a sleep task can just spin. On balance, I'm split.

> <echo message="starting JUnit test"/>
> <java taskname="junit"
> fork="yes"
> classpath="${testrun.classpath}"
> classname="AllTests"/>
> <wlstop classpath="${weblogic.classes}"
> user="..."
> password="..."
> url="..." />
> <join/>

Once again, since this is flow control, maybe a PI? Would the join task look
for the thread that was started? Are all asynch threads registered somewhere
in the project?

> <echo message="test finished"/>
> </target>

Hmmm... Warming... I guess I don't have an issue with doing acutal threading
if it's something like this... I'm really wanting to see us hit a useful and
simple way of expressing it if we are going to do it.

If you look at PI's and decide that I'm smoking dope, then go ahead with
what you've got, I'll remove my -1.

.duncan

RE: [PATCH] multithreading tasks

[Conor MacNeill <co...@cortexebusiness.com.au>]

> -----Original Message-----
> From: James Duncan Davidson [mailto:james.davidson@eng.sun.com]
>
> on 2000/06/18 05:26, Conor MacNeill at conor@m64.com wrote:
>
> Oh, I'm open to being "worked" on. And hopefully my being pig-headish well
> help in the long run. :)

No doubt :-)

>
> Help me out -- give me some syntax to work with. I would like to see a
> sample of the XML build file.

OK, here is my actual test setup with some attributes elided for brevity

<target name="test" depends="install, testcode">
  <wlrun taskname="server"
         async="yes"
         classpath="${weblogic.boot.classpath}"
         wlclasspath="${weblogic.classes}:..."
         name="${server.name}" home="${weblogic.home}"
         properties="${server.name}/${server.name}.properties"/>
  <sleep delay="80000"/>
  <echo message="starting JUnit test"/>
  <java taskname="junit"
        fork="yes"
        classpath="${testrun.classpath}"
        classname="AllTests"/>
  <wlstop classpath="${weblogic.classes}"
          user="..."
          password="..."
          url="..." />
  <join/>
  <echo message="test finished"/>
</target>

Let me explain a few things.

<wlrun> is a task which I use to start a weblogic server (EJB Container).
<wlstop> is a task to stop a weblogic server.

In the patch I sent in last week, I made changes to tag all task logging
with a taskname. This is set to the task type (javac, deltree, etc) by
default but can be overridden by the taskname attribute. Thus I can see the
<java> tasks output as "junit" in the log. Matt's build events patch also
tags all task log output with the task type name.

The wlrun task is run in a separate thread (controlled by the "async"
attribute). The <join>before the echo is for demonstration purposes only -
there is an implicit join performed at the end of the target anyway. The
<sleep> ensures the server has completed initialisation before JUnit starts
to perform its tests.

The output (with much stuff removed)

[server] Thu Jun 22 20:45:33 GMT+10:00 2000:<I> <WebLogicServer> WebLogic
Server started
starting JUnit test
[...]
[junit] Time: 836.332
[junit]
[junit] FAILURES!!!
[junit] Test Results:
[junit] Run: 70 Failures: 3 Errors: 0
[junit] There were 3 failures:
[junit]
test finished
Completed in 1024 seconds

>
> If I understand right, you've got the threading constructs set up as tasks
> themselves. I'm not sure that they are tasks really since they are telling
> Ant to do certain things. Feels a bit like operator overloading
> to me if I'm
> imagining it right. ;) Maybe seeing a build file will help me with this.
>

sleep and join are implemented as tasks but I guess that is equivalent to
those concepts being methods in Java itself. Is that what you meant here?

>
> My main worry is comprehensibility/clarity. Help me with that point and
> let's see where it leads us.
>

It seems to be pretty straight forward to me - whaddaya reckon?

Conor

Re: [PATCH] multithreading tasks

[James Duncan Davidson <ja...@eng.sun.com>]

on 2000/06/18 05:26, Conor MacNeill at conor@m64.com wrote:

> No, this has not been committed yet. Let me see if I can "work" on you to
> see whether it is a good idea or not? Let me try to address each of your
> issues ...

Oh, I'm open to being "worked" on. And hopefully my being pig-headish well
help in the long run. :)

Help me out -- give me some syntax to work with. I would like to see a
sample of the XML build file.

If I understand right, you've got the threading constructs set up as tasks
themselves. I'm not sure that they are tasks really since they are telling
Ant to do certain things. Feels a bit like operator overloading to me if I'm
imagining it right. ;) Maybe seeing a build file will help me with this.
 
> Its part of the "test often" philosophy and I don't have to remember to run
> the tests. If I make it part of the CVS check-in process then it is even
> better.

No argument that test often is the best way to go.

> Why? Perhaps you could expand on the likely problems? I have been using
> parallel tasks now for a while. There is no interaction between the tasks
> within Ant. The external interaction is well defined and interesting :-)

My main worry is comprehensibility/clarity. Help me with that point and
let's see where it leads us.

.duncan

RE: [PATCH] multithreading tasks

[Conor MacNeill <co...@m64.com>]

Duncan,

> From: James Duncan Davidson [mailto:james.davidson@eng.sun.com]
>
> Has this been committed? I didn't see a commit message go out on
> it. In any
> case I'm -1 on this.

No, this has not been committed yet. Let me see if I can "work" on you to
see whether it is a good idea or not? Let me try to address each of your
issues ...

> MT programming isn't *that* easy and ...

I agree in general that MT can have pitfalls. I have less of a problem,
however, with multithreading Ant tasks. If Ant tasks are run concurrently
they will have minimal interaction with each other within Ant. Such
interaction would be through the Project ant Target objects and, in
practice, most tasks do not use those objects extensively. The potential for
interaction outside of Ant (through the file system, for example) is there,
I guess, but again I don't see it as problematic. If there are such
interactions, the tasks should be run sequentially.

> I don't really think that projects should deal with it.

OK. I think there are two cases. Within a target, there may be some tasks
which are independent. Those independent tasks could be executed
concurrently. In this situation, concurrent operation is not essential but
could be useful for performance reasons. An example might be where it is
necessary to download code from two independent CVS repositories. Another
example might be to execute a set of ant subprojects in parallel to build a
set of, independent, jars.

In the other cases, two tasks MUST be executed concurrently because of their
interaction, external to Ant. Because of this interaction, they cannot be
executed sequentially. An example would be running a test client against a
server. This is my particular situation. I am running a JUnit test harness
against a Weblogic EJB container. I can do this outside of Ant, of course,
but I really like the integration of the testing step with the build step.
Its part of the "test often" philosophy and I don't have to remember to run
the tests. If I make it part of the CVS check-in process then it is even
better.

>
> If concurency is needed for testing, then doing a test that that performs
> it's own parallelism at that level is fine, ...

I am not interested in parallelising the tests themselves, if that is what
you mean. I merely need to run the test system at the same time as the
system being tested. These are both standard tasks (I run both as <java>
tasks). If I have to combine these tasks to get the necessary parallelism
happening inside a taskdef, that task would not be useful or reusable. How
many people could use a <WeblogicWithJUnitTest> task? The components are
there in Ant and I want to put them together in a parallel fashion at the
build.xml level and not within a taskdef.

> but building parallelism into Ant gets probelmatic fast.

Why? Perhaps you could expand on the likely problems? I have been using
parallel tasks now for a while. There is no interaction between the tasks
within Ant. The external interaction is well defined and interesting :-)

I understand that simplicity is the Ant motto and the inclusion of this
feature provides a power which could be misused. Nevertheless, on some
occasions I feel that power is needed. It is off by default which means it
requires definite thought to turn it on (OK, maybe not much thought :-)

>
> > Let me know what you think. If this is accepted, I plan two new small
> > taskdefs:- a sleep task and a join task.
>
> Problem here is that now you are using tasks not to perform functionality,
> but as logical control.

OK. Fair point. I perform an implicit join at the end of a target so the
join task could be left out and the same effect achieved by target
dependency relationships. My motivation was to be able to bring the threads
back together within the single target and then continue sequentially.
Creating a whole set of targets and dependencies seemed to reduce the
cohesiveness of the targets.

Sleep is really a simple form of task coordination so I can see your
objection. Nevertheless, I have found it useful, since weblogic takes a
while to start up and deploy EJBs. I have to wait for that to happen before
I start my JUnit test harness. I didn't want any more complicated task
coordination so sleep suffices.

> Expressing threading semantics in the XML is neat, but complicates
> things to a great degree.
>

Why?

> I understand the motive -- parallelism on tests is a good thing,
> but I don't
> see why that functionality has to be genericized to this level.
>

OK, I've made my arguments. Perhaps you and others can comment?

Cheers

Conor

Re: [PATCH] multithreading tasks

[James Duncan Davidson <ja...@eng.sun.com>]

on 5/28/00 8:58 AM, Conor MacNeill at conor@m64.com wrote:

> Attached is a fairly simple patch plus a new class to allow ant to run tasks
> in separate threads. The motivation for this patch is to allow us to perform
> unit testing in a relatively automated fashion. In our particular situation
> we would like to have an ant target for testing which will start weblogic in
> one thread and JUnit in another. Once JUnit completes its tests, weblogic
> will be stopped and the target will complete by joining all outstanding
> threads.

Has this been committed? I didn't see a commit message go out on it. In any
case I'm -1 on this.

If concurency is needed for testing, then doing a test that that performs
it's own parallelism at that level is fine, but building parallelism into
Ant gets probelmatic fast. MT programming isn't *that* easy and I don't
really think that projects should deal with it.

> Let me know what you think. If this is accepted, I plan two new small
> taskdefs:- a sleep task and a join task.

Problem here is that now you are using tasks not to perform functionality,
but as logical control. Sam has worked on me enough that I'm no longer
totally freaked out about layering scripting into Ant, as long as that
scripting isn't expressed in the XML. Expressing threading semantics in the
XML is neat, but complicates things to a great degree.

I understand the motive -- parallelism on tests is a good thing, but I don't
see why that functionality has to be genericized to this level.

.duncan