[PATCH] Limit amount of parallelism for apply task

["Frank A. Hunleth" <fh...@cs.wustl.edu>]

Hi,

I'm a daily user of ant, but this is my first submission back.  Let me
know if there's anything else that you need for this patch.

The problem that I ran into was that I was running a program using the
apply task.  It ran faster if given many files to work on in parallel, but
in one situation the number of files passed to it overwhelmned it.  A
coworking of mine thinks that I may have exceeded a command line length
limit of 64K and that the program wasn't to blame.  In any case, running
the program in serial takes painfulling long, so what I really wanted to
do was run, say, 50 source files at a time.

I updated ExecuteOn.java to do this, and the patch is included below.  
Basically I added a new attribute, parallelamount, to allow one to specify
the maximum number of files that should be operated on in parallel at a
time.  It only gets used if the parallel attribute is set to true.

Thanks,
Frank


Index: ExecuteOn.java
===================================================================
RCS
file: /home/cvspublic/jakarta-ant/src/main/org/apache/tools/ant/taskdefs/ExecuteOn.java,v
retrieving revision 1.19
diff -u -b -u -r1.19 ExecuteOn.java
--- ExecuteOn.java	2001/11/21 17:29:43	1.19
+++ ExecuteOn.java	2001/12/03 18:35:52
@@ -80,6 +80,7 @@
     protected Vector filesets = new Vector();
     private boolean relative = false;
     private boolean parallel = false;
+    private int parallelAmount = 0;
     protected String type = "file";
     protected Commandline.Marker srcFilePos = null;
     private boolean skipEmpty = false;
@@ -116,6 +117,14 @@
     }
 
     /**
+     * If parallel is true, then up to how many files can be processed
+     * at a time? ( <= 0 indicates no upper bound)
+     */
+    public void setParallelAmount(int amount) {
+        this.parallelAmount = amount;
+    }
+
+    /**
      * Shall the command work only on files, directories or both?
      */
     public void setType(FileDirBoth type) {
@@ -199,6 +208,42 @@
         }
     }
 
+    protected void runInParallel(Execute exe,
+                                 Vector fileNames,
+                                 Vector baseDirs)
+        throws BuildException, IOException {
+        String[] s = new String[fileNames.size()];
+        fileNames.copyInto(s);
+        File[] b = new File[baseDirs.size()];
+        baseDirs.copyInto(b);
+
+        if (this.parallelAmount <= 0) {
+            String[] command = getCommandline(s, b);
+            log("Executing " + Commandline.toString(command),
+                Project.MSG_VERBOSE);
+            exe.setCommandline(command);
+            runExecute(exe);
+        } else {
+            int amountLeft = fileNames.size();
+            int currentOffset = 0;
+            while (amountLeft > 0) {
+                int currentAmount = Math.min(amountLeft,
this.parallelAmount);
+                String[] cs = new String[currentAmount];
+                System.arraycopy(s, currentOffset, cs, 0, currentAmount);
+                File[] cb = new File[currentAmount];
+                System.arraycopy(b, currentOffset, cb, 0, currentAmount);
+                String[] command = getCommandline(cs, cb);
+                log("Executing " + Commandline.toString(command),
+                    Project.MSG_VERBOSE);
+                exe.setCommandline(command);
+                runExecute(exe);
+
+                amountLeft -= currentAmount;
+                currentOffset += currentAmount;
+            }
+        }
+    }
+
     protected void runExec(Execute exe) throws BuildException {
         try {
 
@@ -247,15 +292,7 @@
             }
 
             if (parallel && (fileNames.size() > 0 || !skipEmpty)) {
-                String[] s = new String[fileNames.size()];
-                fileNames.copyInto(s);
-                File[] b = new File[baseDirs.size()];
-                baseDirs.copyInto(b);
-                String[] command = getCommandline(s, b);
-                log("Executing " + Commandline.toString(command), 
-                    Project.MSG_VERBOSE);
-                exe.setCommandline(command);
-                runExecute(exe);
+                runInParallel(exe, fileNames, baseDirs);
             }
 
         } catch (IOException e) {



--
To unsubscribe, e-mail:   <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>

Re: [PATCH] Limit amount of parallelism for apply task

[Stefan Bodewig <bo...@apache.org>]

On Mon, 3 Dec 2001, Frank A. Hunleth <fh...@cs.wustl.edu> wrote:

> The problem that I ran into was that I was running a program using
> the apply task.  It ran faster if given many files to work on in
> parallel, but in one situation the number of files passed to it
> overwhelmned it.

Thanks, the patch finally made it into Ant's CVS (almost eighteen
months on my TODO list, sorry).

Stefan