asynchronous tasks

[Conor MacNeill <co...@m64.com>]

Hi,

A while back I posted a patch to allow tasks to be run in a separate thread.
I have made a few other changes now which I would like to run by everyone.
Whilst I made these changes primarily to support async operation, they do
affect the normal operation of ant.

When running multiple threads it is important to distinguish the output of
separate threads. I have therefore introduced a task name which defaults to
the task type but which can be set explicitly in the task XML. I have
created log methods in the Task class which add this task name into log
messages. These call the appropriate Project.log methods, with the taskname
as a tag. (Many tasks were doing this anyway). The resulting output would be
something like

[mkdir] Created dir: F:\Projects\jakarta\build\ant\classes
[javac] Compiling 56 source files to F:\Projects\jakarta\build\ant\classes
[javac] Copying 2 support files to F:\Projects\jakarta\build\ant\classes
Executing Target: jar
[jar] Building jar: F:\Projects\jakarta\jakarta-ant\lib\ant.jar
Executing Target: javadocs
[mkdir] Created dir: F:\Projects\jakarta\build\ant\javadocs
[javadoc] Generating Javadoc
[javadoc err] javadoc: warning - Import not found: netrexx.lang.Rexx -
ignoring!
[javadoc] 3 warnings
Executing Target: dist
[mkdir] Created dir: F:\Projects\jakarta\dist\ant
[copydir] Copying 60 files to F:\Projects\jakarta\dist\ant\src

I am also including two new taskdefs used in multithreaded situations
<sleep> and <join>. Sleep simply sleeps for some milliseconds and join joins
all current task threads to the main ant thread.

The diffs and new classes are attached.

If no one objects to these changes, I will commit them next week.

Cheers
Conor



--
Conor MacNeill
Home: conor@m64.com
Work: conor@cortexebusiness.com.au
Web:  www.cortexebusiness.com.au

Re: asynchronous tasks

[Peter Donald <do...@mad.scientist.com>]

At 05:03  17/6/00 +0200, you wrote:
>Hi Conor,
>
>What is the advantage of executing tasks asynch ? Perhaps I'm missing
>something here, but I'd say they are usually processor-bound, so you will not
>get extra speed out of it. And since a Java process is single-process by
>design, I assume even a second processor would not be able to speed this up.

It's extremely useful if you have to transfer stuff across the network.
None of my tasks need this functionality now but I suspect they may in the
future 

Cheers,

Pete

*------------------------------------------------------*
| "Nearly all men can stand adversity, but if you want |
| to test a man's character, give him power."          |
|       -Abraham Lincoln                               |
*------------------------------------------------------*

RE: asynchronous tasks

[Tom Dimock <ta...@cornell.edu>]

At 08:08 PM 6/17/00 -0400, Rick Yazwinski wrote:
>> And since a Java process is
>> single-process by
>> design, I assume even a second processor would not be able to
>> speed this
>> up.
>
>Wouldn't this depend on the architecture's implementation of threads?  I'd
>expect on a truely multitasking box (Solaris or Linux, for example) with
>multiple processors that you would see a performance improvement.

I was also mystified by this comment, as my dual processor Dell box running
Windows NT will definitely use both processors, which gives VERY noticible
performance improvements - and also uncovers a lot of bugs in
multi-threaded code which was written on a single-processor machine  :-(
I'm not sure where the original poster got the idea that Java is "single
process by design".

One of the problems of a naive design which spawns lots of threads is that
it can hog all of the processors even on a large server with 32 or more
processors.  One of the more subtle reasons to use thread pooling is to
control this kind of anti-social behavior.

Re: asynchronous tasks

[Sean Brandt <se...@fuzzymagic.com>]

A multithreaded app in a single JVM will benefit from multi-cpu if the jvm
implements threads in a light weight manner. NT, Solaris, Linux, *BSD all have
lightweight kernel threads, so that a single VM can make use of multiple
processors. Now, are there some cases where kernel threads are not as well done as
OS taskswitching, maybe (Those OS's to be left nameless) ;)

- Sean

Ernst de Haan wrote:

> Hi Conor,
>
> What is the advantage of executing tasks asynch ? Perhaps I'm missing
> something here, but I'd say they are usually processor-bound, so you will not
> get extra speed out of it. And since a Java process is single-process by
> design, I assume even a second processor would not be able to speed this up.
>
> Ernst
>
> Conor MacNeill wrote:
> > Hi,
> >
> > A while back I posted a patch to allow tasks to be run in a separate thread.
> > I have made a few other changes now which I would like to run by everyone.
> > Whilst I made these changes primarily to support async operation, they do
> > affect the normal operation of ant.
> >
> > When running multiple threads it is important to distinguish the output of
> > separate threads. I have therefore introduced a task name which defaults to
> > the task type but which can be set explicitly in the task XML. I have
> > created log methods in the Task class which add this task name into log
> > messages. These call the appropriate Project.log methods, with the taskname
> > as a tag. (Many tasks were doing this anyway). The resulting output would be
> > something like
> >
> > [mkdir] Created dir: F:\Projects\jakarta\build\ant\classes
> > [javac] Compiling 56 source files to F:\Projects\jakarta\build\ant\classes
> > [javac] Copying 2 support files to F:\Projects\jakarta\build\ant\classes
> > Executing Target: jar
> > [jar] Building jar: F:\Projects\jakarta\jakarta-ant\lib\ant.jar
> > Executing Target: javadocs
> > [mkdir] Created dir: F:\Projects\jakarta\build\ant\javadocs
> > [javadoc] Generating Javadoc
> > [javadoc err] javadoc: warning - Import not found: netrexx.lang.Rexx -
> > ignoring!
> > [javadoc] 3 warnings
> > Executing Target: dist
> > [mkdir] Created dir: F:\Projects\jakarta\dist\ant
> > [copydir] Copying 60 files to F:\Projects\jakarta\dist\ant\src
> >
> > I am also including two new taskdefs used in multithreaded situations
> > <sleep> and <join>. Sleep simply sleeps for some milliseconds and join joins
> > all current task threads to the main ant thread.
> >
> > The diffs and new classes are attached.
> >
> > If no one objects to these changes, I will commit them next week.
> >
> > Cheers
> > Conor
> >
> >
> >
> > --
> > Conor MacNeill
> > Home: conor@m64.com
> > Work: conor@cortexebusiness.com.au
> > Web:  www.cortexebusiness.com.au

Re: asynchronous tasks

[Steve Loughran <st...@iseran.com>]

----- Original Message -----
From: "Mark Brouwer" <ma...@virgil.nl>
To: <an...@jakarta.apache.org>
Sent: Wednesday, June 21, 2000 23:36
Subject: Re: asynchronous tasks


>
>
> James Duncan Davidson wrote:
> >
> > on 2000/06/17 17:08, Rick Yazwinski at ricky@solect.com wrote:
> >
> > > Wouldn't this depend on the architecture's implementation of threads?
I'd
> > > expect on a truely multitasking box (Solaris or Linux, for example)
with
> > > multiple processors that you would see a performance improvement.
> >
> > Except that javac runs in a single thread I beleive. You can span a
thread
> > across processors. :)
> >
>
> Correct, however if have never seen the processor load for a single
> javac task above 100/n %, where n is the number of processors on my Sun
> E420

I was thinking last night about how to run javac task do as to get extra
work done on a multiway system. The problem for me is that unlike the simple
C/C++ dependency pipeline, javac uses the outputs of javac (.class files) as
the input to javac. So it would seem unwise to split every compilation of
every file into a separate thread, even if you had CPUs to spare.

You could spin off the single javac thread and run it on its own -and have
it tickle some synchronisation object when it had finished. A later task
could depend not only on the javac task but block until the sync object was
signalled.

A more devious trick would be to have the javac execution component hooked
to ant by a synchronised queue down which build requests were sent, so that
the compile task would just pump off a list of files to build (and a
reference to a sync object). The execution component could then handle
requests from multiple tasks, keeping it busy.

But my build problems arent complex enough to merit any of this work.
Running two different ants on two different build.xml files in two different
windows is adequate multithreading for me.

    -Steve

Re: asynchronous tasks

[Steve Loughran <st...@iseran.com>]

----- Original Message -----
From: "Mark Brouwer" <ma...@virgil.nl>
To: <an...@jakarta.apache.org>
Sent: Wednesday, June 21, 2000 23:36
Subject: Re: asynchronous tasks


>
>
> James Duncan Davidson wrote:
> >
> > on 2000/06/17 17:08, Rick Yazwinski at ricky@solect.com wrote:
> >
> > > Wouldn't this depend on the architecture's implementation of threads?
I'd
> > > expect on a truely multitasking box (Solaris or Linux, for example)
with
> > > multiple processors that you would see a performance improvement.
> >
> > Except that javac runs in a single thread I beleive. You can span a
thread
> > across processors. :)
> >
>
> Correct, however if have never seen the processor load for a single
> javac task above 100/n %, where n is the number of processors on my Sun
> E420

I was thinking last night about how to run javac task do as to get extra
work done on a multiway system. The problem for me is that unlike the simple
C/C++ dependency pipeline, javac uses the outputs of javac (.class files) as
the input to javac. So it would seem unwise to split every compilation of
every file into a separate thread, even if you had CPUs to spare.

You could spin off the single javac thread and run it on its own -and have
it tickle some synchronisation object when it had finished. A later task
could depend not only on the javac task but block until the sync object was
signalled.

A more devious trick would be to have the javac execution component hooked
to ant by a synchronised queue down which build requests were sent, so that
the compile task would just pump off a list of files to build (and a
reference to a sync object). The execution component could then handle
requests from multiple tasks, keeping it busy.

But my build problems arent complex enough to merit any of this work.
Running two different ants on two different build.xml files in two different
windows is adequate multithreading for me.

    -Steve

Re: asynchronous tasks

[James Duncan Davidson <ja...@eng.sun.com>]

on 2000/06/21 23:36, Mark Brouwer at mark.brouwer@virgil.nl wrote:

> James Duncan Davidson wrote:
>> Except that javac runs in a single thread I beleive. You can span a thread
>> across processors. :)

Er. I meant to say *can't*. :(

Re: asynchronous tasks

[Mark Brouwer <ma...@virgil.nl>]

James Duncan Davidson wrote:
> 
> on 2000/06/17 17:08, Rick Yazwinski at ricky@solect.com wrote:
> 
> > Wouldn't this depend on the architecture's implementation of threads?  I'd
> > expect on a truely multitasking box (Solaris or Linux, for example) with
> > multiple processors that you would see a performance improvement.
> 
> Except that javac runs in a single thread I beleive. You can span a thread
> across processors. :)
> 

Correct, however if have never seen the processor load for a single
javac task above 100/n %, where n is the number of processors on my Sun
E420

> Note that I'm not for mt ant at all.
> 
> > Has anyone brought up the potential of "distributed ant"?  It kinda fits
> > with the async talk...
> 
> Distrubted ant? Complexity overload anyone?
> 

Well I still need a JavaSpaces project to start with, now I can finally
implement the 'replicated-worker pattern', so if you can arrange the
funding :-) 

> .duncan

-- 
Mark Brouwer

Re: asynchronous tasks

[James Duncan Davidson <ja...@eng.sun.com>]

on 2000/06/17 17:08, Rick Yazwinski at ricky@solect.com wrote:

> Wouldn't this depend on the architecture's implementation of threads?  I'd
> expect on a truely multitasking box (Solaris or Linux, for example) with
> multiple processors that you would see a performance improvement.

Except that javac runs in a single thread I beleive. You can span a thread
across processors. :)

Note that I'm not for mt ant at all.

> Has anyone brought up the potential of "distributed ant"?  It kinda fits
> with the async talk...

Distrubted ant? Complexity overload anyone?

.duncan

RE: asynchronous tasks

[Rick Yazwinski <ri...@solect.com>]

> And since a Java process is
> single-process by
> design, I assume even a second processor would not be able to
> speed this
> up.

Wouldn't this depend on the architecture's implementation of threads?  I'd
expect on a truely multitasking box (Solaris or Linux, for example) with
multiple processors that you would see a performance improvement.

Has anyone brought up the potential of "distributed ant"?  It kinda fits
with the async talk...

	Rick...

RE: asynchronous tasks

[Conor MacNeill <co...@m64.com>]

Ernst,

>
> Hi Conor,
>
> What is the advantage of executing tasks asynch ? Perhaps I'm missing
> something here, but I'd say they are usually processor-bound, so
> you will not
> get extra speed out of it. And since a Java process is single-process by
> design, I assume even a second processor would not be able to
> speed this up.
>

Performance of builds is not the motivation here. We use ant to perform a
JUnit test run on our system, a Weblogic application. I start a Weblogic
server in one ant task and run JUnit in another. These tasks need to run
concurrently. Nevertheless, our builds do take a fair while and I think it
may be possible to speed this up by using concurrent processes.

Conor

Re: asynchronous tasks

[Ernst de Haan <er...@jollem.com>]

Hi Conor,

What is the advantage of executing tasks asynch ? Perhaps I'm missing
something here, but I'd say they are usually processor-bound, so you will not
get extra speed out of it. And since a Java process is single-process by
design, I assume even a second processor would not be able to speed this up.


Ernst


Conor MacNeill wrote:
> Hi,
> 
> A while back I posted a patch to allow tasks to be run in a separate thread.
> I have made a few other changes now which I would like to run by everyone.
> Whilst I made these changes primarily to support async operation, they do
> affect the normal operation of ant.
> 
> When running multiple threads it is important to distinguish the output of
> separate threads. I have therefore introduced a task name which defaults to
> the task type but which can be set explicitly in the task XML. I have
> created log methods in the Task class which add this task name into log
> messages. These call the appropriate Project.log methods, with the taskname
> as a tag. (Many tasks were doing this anyway). The resulting output would be
> something like
> 
> [mkdir] Created dir: F:\Projects\jakarta\build\ant\classes
> [javac] Compiling 56 source files to F:\Projects\jakarta\build\ant\classes
> [javac] Copying 2 support files to F:\Projects\jakarta\build\ant\classes
> Executing Target: jar
> [jar] Building jar: F:\Projects\jakarta\jakarta-ant\lib\ant.jar
> Executing Target: javadocs
> [mkdir] Created dir: F:\Projects\jakarta\build\ant\javadocs
> [javadoc] Generating Javadoc
> [javadoc err] javadoc: warning - Import not found: netrexx.lang.Rexx -
> ignoring!
> [javadoc] 3 warnings
> Executing Target: dist
> [mkdir] Created dir: F:\Projects\jakarta\dist\ant
> [copydir] Copying 60 files to F:\Projects\jakarta\dist\ant\src
> 
> I am also including two new taskdefs used in multithreaded situations
> <sleep> and <join>. Sleep simply sleeps for some milliseconds and join joins
> all current task threads to the main ant thread.
> 
> The diffs and new classes are attached.
> 
> If no one objects to these changes, I will commit them next week.
> 
> Cheers
> Conor
> 
> 
> 
> --
> Conor MacNeill
> Home: conor@m64.com
> Work: conor@cortexebusiness.com.au
> Web:  www.cortexebusiness.com.au