You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uima.apache.org by ch...@apache.org on 2013/09/21 00:29:14 UTC

svn commit: r1525159 - in /uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4: admin/ducc-classes.tex rm.tex sim.tex

Author: challngr
Date: Fri Sep 20 22:29:13 2013
New Revision: 1525159

URL: http://svn.apache.org/r1525159
Log:
UIMA-2682 Clean up RM and Sim sections.

Modified:
    uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/admin/ducc-classes.tex
    uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/rm.tex
    uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/sim.tex

Modified: uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/admin/ducc-classes.tex
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/admin/ducc-classes.tex?rev=1525159&r1=1525158&r2=1525159&view=diff
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/admin/ducc-classes.tex (original)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/admin/ducc-classes.tex Fri Sep 20 22:29:13 2013
@@ -175,6 +175,7 @@ ducc.classes and is specified by the pro
 
 
     \subsection{Class Definitions}
+    \label{subsubsec:class.configuration}
 
     Scheduler classes are defined in the same simple block language as
     nodepools.

Modified: uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/rm.tex
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/rm.tex?rev=1525159&r1=1525158&r2=1525159&view=diff
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/rm.tex (original)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/rm.tex Fri Sep 20 22:29:13 2013
@@ -13,28 +13,26 @@
     \begin{description}
         \item[Managed Jobs]
             Managed jobs are Java applications implemented in the UIMA framework. 
-            They are scaled out by DUCC using UIMA-AS. Managed jobs are executed as some 
+            and are scaled out by DUCC using UIMA-AS. Managed jobs are executed as some 
             number of discrete processes distributed over the cluster resources. All processes of all jobs 
             are by definition preemptable; the number of processes is allowed to increase and decrease 
             over time in order to provide all users access to the computing resources. 
         \item[Services]
             Services are long-running processes which perform some function on behalf of 
-            jobs or other services. Most DUCC services are UIMA-AS services and are managed the 
-            same as managed jobs. From a scheduling point of view, there is no difference between 
-            services and managed jobs. 
+            jobs or other services. Most DUCC services are UIMA-AS assigned to a non-preemptable 
+            resource class, as defined below.
 
         \item{Reservations}
-            A reservation provides persistent, dedicated use of some portion of the 
-            resources to a specific user. A reservation may be for an entire machine, or it may be for 
-            some portion of a machine. Machines are subdivided according to the amount of memory 
-            installed on the machine. 
-            The work that DUCC is designed to support is extremely memory-intensive. In most cases 
-            resources are significantly more constrained by memory than by CPU processing poser. The entire 
-            resource pool in a DUCC cluster therefore consists of the total memory of all the processors in the           
-            cluster. 
+            A reservation provides non-preemptable, persistent, dedicated use of a full machine or
+            some part of a machine to a specific user.
 
         \item{Arbitrary Processes}
-            \todo Define these.
+            An {\em arbitrary process} or {\em managed reservation} is any process at all, which may
+            or may not have anything to do with UIMA.  These processes are usually used for services,
+            or to launch very large Eclipse work-spaces for debugging.  DUCC supports this type of 
+            process but is not optimized for it.  These processes are usually scheduled to be 
+            non-preemptable, occupying either a dedicated machine or some portion of a machine.
+
       \end{description}
           
     In order to apportion the cumulative memory resource among requests, the Resource Manager 
@@ -72,9 +70,9 @@
     an equal number of shares. This weighed adjustment of fair-share assignments is called weighted 
     fair share. 
     
-    The abstraction used to organized jobs by importance is the job class or simply class. As jobs enter 
+    The abstraction used to organized jobs by importance is the job class or simply ``class''. As jobs enter 
     the system they are grouped with other jobs of the same importance and assigned to a common 
-    class. The class and its attributes are described in subsequent sections. 
+    class. The class abstraction and its attributes are described in \hyperref[sec:rm.job-classes]{subsequent sections}. 
     
     The scheduler executes in two phases: 
     \begin{enumerate}
@@ -98,16 +96,19 @@
           user may have jobs in more than one class, in which case that user's fair share is calculated
           independently within each class.
           
-        \item Job counts: For each user (independently within each class), collect all the jobs
-          assigned to that user and apply fair-share to equally divide all the user's shares among
-          their jobs. This apportions all shares given to this user for each class among the user's
-          jobs in that class.  Reservations are relatively simple. If the number of shares or
-          machines requested is available or can be made available through preemption of fair-share
-          jobs, the reservation is satisfied and resources are allocated. If not, the reservation
-          fails. In the case where preemptions are required, the reservation is delayed until all
-          necessary resources have been freed.
+        \item Job counts: For each user, collect all jobs
+          assigned to that user and equally divide all the user's shares among
+          the jobs. This apportions all shares given to this user for each class among the user's
+          jobs in that class. 
     \end{enumerate}
 
+    Reservations are relatively simple. If the number of shares or machines requested is available
+    the reservation succeeds immediately.  If a sufficient number of co-located shares can be made
+    available through preemption of fair-share jobs, preemptions are scheduled and the reservation
+    is deferred until space becomes available.  resources are allocated. If space cannot be found by
+    means of preemption, the reservation fails.
+
+
     \section{Scheduling Policies}
 
     The Resource Manager implements three coexistent scheduling policies. 
@@ -115,22 +116,39 @@
         \item[FAIR\_SHARE] This is the weighted-fair-share policy described in detail above.
 
         \item[FIXED\_SHARE] The FIXED\_SHARE policy is used to reserve a portion of a machine. The
-          allocation is treated as a reservation in that it is permanently allocated (until it is
-          canceled) and it cannot be preempted by any other request.
+          allocation is non-preemptable and remains active until it is canceled.
 
-          A fixed-share request specifies a number of processes of a given size, for example, "10 
+          FIXED\_SHARE allocations have several uses:
+          \begin{itemize}
+            \item As reservations.  In this case DUCC starts no work in the share(s); the user must
+              log in (or run something via ssh), and then manually release the reservation to free
+              the resources.  This is often used for testing and debugging.
+            \item For services.  If a service is registered to run in a FIXED\_SHARE allocation,
+              DUCC allocates the resources, starts and manages the service, and releases the
+              resource if the service is stopped or unregistered.
+            \item For UIMA jobs.  A ``normal'' UIMA job may be submitted to a FIXED\_SHARE
+              class.  In this case, the processes are never preempted, allowing constant and
+              predictable execution of the job.  The resources are automatically released when
+              the job exits.
+          \end{itemize}
+          
+          {\em Note:} A fixed-share request specifies a number of processes of a given size, for example, "10 
           processes of 32GB each". The ten processes may or may not be collocated on the same 
           machine. Note that the resource manager attempts to minimize fragmentation so if there is a 
           very large machine with few allocations, it is likely that there will be some collocation of the 
           assigned processes. 
           
-          A fixed-share allocation may be thought of a reservation for a "partial" machine. 
-
+          {\em Note:} A fixed-share allocation may be thought of a reservation for a "partial" machine. 
+          
         \item[RESERVE] The RESERVE policy is used to reserve a full machine. It always returns an
           allocation for an entire machine. The reservation is permanent (until it is canceled) and
           it cannot be preempted by any other request.
 
-          It is possible to configure the scheduling policy so that a reservation returns any machine in 
+          Reservations may also be used for services or UIMA jobs in the same way as FIXED\_SHARE 
+          allocations, the difference being that a reservation occupies full machines and FIXED\_SHARE
+          occupies portions of a machine.
+
+          {\em Note:} It is possible to configure the scheduling policy so that a reservation returns any machine in 
           the cluster that is available, or to restrict it to machines of the size specified in the reservation 
           request. 
     \end{description}
@@ -142,20 +160,21 @@
     high importance jobs. Priorities are used to resolve these conflicts
 
     Simply: priority is used to specify the order of evaluation of the job classes. Weight is used
-    to specify the importance (or weights) of the job classes for use by the weighted fair-share
-    scheduling policy.
+    to proportionally allocate the number of shares to each class under the weighted fair-share
+    policies.
 
     \paragraph{Priority.} It is possible that conflicts may arise in scheduling policies. For example, it may be
     desired that reservations be fulfilled before any fair-share jobs are scheduled. It may be
     desired that some types of jobs are so important that when they enter the system all other
     fair-share jobs be evicted. Other such examples can be found.
     
-    To help resolve this, the Resource Manager allows job classes to be prioritized. Priority is
+    To resolve this, the Resource Manager allows job classes to be prioritized. Priority is
     used to determine the order of evaluation of the scheduling classes.
     
     When a scheduling cycle starts, the scheduling classes are ordered from "best" to "worst" priority. 
     The scheduler then attempts to allocate ALL of the system's resources to the "best" priority class. 
-    If any resources are left, the scheduler goes on to the next class and so on, until either all the 
+    If any resources are left, the scheduler proceeds to schedule classes in the next best
+    priority, and so on, until either all the 
     resources are exhausted or there is no more work to schedule. 
     
     It is possible to have multiple job classes of the same priority. What this means is that resources 
@@ -165,16 +184,13 @@
     
     To constrain high priority jobs from completely monopolizing the system, class caps may be 
     assigned. Higher priority guarantees that some resources will be available (or made available) but 
-    doesn't that that all resources necessarily be used. 
+    doesn't require that all resources necessarily be used. 
 
     \paragraph{Weight.} Weight is used to determine the relative importance of jobs in a set of job classes of 
     the same priority when doing fair-share allocation. All job classes of the same priority are assigned 
     shares from the full set of available resources according to their weights using weighted fair-share. 
     Weights are used only for fair-share allocation. 
     
-    Class caps may also be used to insure that very high importance jobs cannot fully monopolize all of 
-    the resources in the system. 
-
     \section{Node Pools}
     It may be desired or necessary to constrain certain types of resource allocations to a specific
     subset of the resources. Some nodes may have special hardware, or perhaps it is desired to
@@ -183,35 +199,28 @@
 
     Nodepools impose hierarchical partitioning on the set of available machines. A nodepool is a
     subset of the full set of machines in the cluster. Nodepools may not overlap. A nodepool may
-    itself contain non-overlapping subpools. The highest level nodepool is called the "global"
-    nodepool. If a job class does not have an associated nodepool, the global nodepool is implicitly
-    associated with he class.
+    itself contain non-overlapping subpools.  It is possible to define and schedule work to
+    multiple, independent nodepools.
 
-    Nodepools are associated with job classes. During scheduling, a job may be assigned resources
+    Job classes are associated with nodepools. During scheduling, a job may be assigned resources
     from its associated nodepool, or from any of the subpools which divide the associated nodepool.
     The scheduler attempts to fully exhaust resources in the associated nodepool before allocating
     within the subpools, and during eviction, attempts to first evict from the subpools. The
     scheduler insures that the nodepool mechanism does not disrupt fair-share allocation.
-
-    If it is desired that jobs assigned to some subpool take priority over jobs that have spilled
-    over from the "superpool", then the class associated with the subpool should be given greater
-    weight, or greater priority, as appropriate. (See the Weight vs Priority discussion.)
-
-    There is no explicit priority associated with nodepools. However, it is possible to assign a
-    "preference" to a specific nodepool, if it is desired that those nodes be chosen first when the
-    are available. Use the nodepool configurations "order" directive to do this.
+    
+    More information on nodepools and their configuration can be \hyperref[subsec:nodepools]{found here}.
 
     \section{Job Classes}
     \label{sec:rm.job-classes}
     The primary abstraction to control and configure the scheduler is the class. A class is simply a set 
-    of rules used to parameterize how resources are assigned to jobs. Every job that enters the system is 
-    associated with one job class. 
+    of rules used to parametrize how resources are assigned to jobs. Every job that enters the system is 
+    associated with a single job class. 
     
     The job class defines the following rules: 
     
     \begin{description}
         \item[Priority] This is the order of evaluation and assignment of resources to this class. See
-          the discussion ofPriority vs Weight for details. 
+          the discussion of priority vs Weight for details. 
 
         \item[Weight] This defines the "importance" of jobs in this class and is used in the weighted
           fair-share calculations. 
@@ -219,7 +228,7 @@
         \item[Scheduling Policy] This defines the policy, fair share, fixed share, or reserve used to
           schedule the jobs in this class.
 
-        \item[Caps] Class caps limit the total resources assigned to a class. This is designed to prevent
+        \item[Cap] Class caps limit the total resources assigned to a class. This is designed to prevent
           high importance and high priority job classes from fully monopolizing the resources. It can be
           used to limit the total resources available to lower importance and lower priority classes.
 
@@ -273,3 +282,4 @@
           classes for either policy.
     \end{description}
         
+    More information on nodepools and their configuration can be \hyperref[subsubsec:class.configuration]{found here}.

Modified: uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/sim.tex
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/sim.tex?rev=1525159&r1=1525158&r2=1525159&view=diff
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/sim.tex (original)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-duccdocs/src/site/tex/duccbook/part4/sim.tex Fri Sep 20 22:29:13 2013
@@ -12,18 +12,18 @@
     DUCC is shipped with support for simulating large clusters of arbitrarily 
     configured nodes.  A simple control file describes some number of
     simulated nodes of arbitrary memory sizes.  DUCC's design allows multiples
-    of these to be spawned on a single node, or on a group of nodes with 
+    of these to be spawned on a single node, or on a small set of nodes with 
     multiple simulated nodes apiece.  The standard testing configuration 
-    used for most of the original developemnt of DUCC consisted of four
+    used for most of the development of DUCC consisted of four
     physical 32-GB machines support 52 simulated nodes of varying memory
     sizes from 32 to 128-GB each.
 
-    To simulate job loads, a simple job that sleeps for some easily configured
+    To simulate job loads, a simple UIMA-AS job that sleeps for some easily configured
     length of time was constructed.  Another control file is used to
     generate \hyperref[sec:cli.ducc-submit]{job specifications} requesting randomly-chosen
     job parameters such as memory requirements, service dependencies and so on.
 
-    The test suite contains a simple AnalysisEngine called
+    The test suite contains a simple Analysis Engine called
     {\tt FixedSleepAE}, and a simple Collection Reader called
     {\tt FixedSleepCR}.  The CR reads a set of sleep times, creates
     CASs, and ships them to the AEs via DUCC's Job Driver.  The CAS
@@ -42,10 +42,9 @@
     workloads and have shown to be very robust for proving the correctness
     of the DUCC code under stress.
 
-    The cluster simulator has been run on a 4GB iMac with 8 simulated Agents, an 8GB MacBook with
-    the same configuration, a 32GB iMac with upt to 40 simulated Agents.  It is usually run on a
-    (shared) test cluster of 4 32GB nodes running Linux and 50-100 simulated nodes, with no
-    performance problems.
+    The cluster simulator has been also been run on a 4GB iMac with 8 simulated Agents, an 8GB MacBook with
+    the same configuration, a 32GB iMac with up to 40 simulated Agents. It has also been scaled
+    up to run on 8 45G Intel nodes running Linux, simulating 20TB of memory.
 
     The rest of this chapter describes the mechanics of using these tools.
 
@@ -59,7 +58,7 @@
 
     DUCC's design divorces all the DUCC processes from specific IP
     addresses or node names.  ActiveMq is used as a nameserver and
-    packet router so that all messages can be deliverd by name,
+    packet router so that all messages can be delivered by name,
     irrespective of the physical hardware the destination process
     may reside upon.  
 
@@ -90,19 +89,19 @@
     
     Normally, the DUCC Agents use the name and IP address of the node 
     they actually do reside upon. This is simply for convenience. 
-    It is possible to parameterize the DUCC Agents to report any arbitrary
+    It is possible to parametrize the DUCC Agents to report any arbitrary
     name and address to the DUCC.  DUCC components that need to know
     about Node Agents establish subscriptions to the Agent publications
     with ActiveMQ and build up their internal structures from the 
     node identities in the Agent publications.  Processes which normally 
-    establish listeners are are the RM, PM, and WS.
+    establish agent listeners are are the RM, PM, and WS.
 
-    It is also possible to parameterize a DUCC agent to cause it to
+    It is also possible to parametrize a DUCC agent to cause it to
     report any arbitrary memory size.  Thus, an agent running on a
     2GB machine can be started so that it reports 32GB of memory. This
-    parameterization is specifically for testing, of course.
+    parametrization is specifically for testing, of course.
 
-    The ability to parameterize agent identities and memory sizes is what enables 
+    The ability to parametrize agent identities and memory sizes is what enables 
     cluster simulation.  A control file is used by start-up scripting
     to spawn multiple agents per node, each with unique identities. 
 
@@ -162,7 +161,7 @@ agentn.79 = 6
       DUCC provides a start-up script in the directory {\tt \duccruntime/examples/systemtest} 
       called {\tt start\_sim}.  
 
-      NOTE: Cluster simulation is intended for DUCC testing, including error injectipon.  It is
+      WARNING: Cluster simulation is intended for DUCC testing, including error injection.  It is
       similar to flying a high-performance fighter jet.  It is intentionally twitchy.  Very little
       checking is done and processes may be started multiple time whether it normally is sane to
       do this.
@@ -180,7 +179,7 @@ agentn.79 = 6
         \item[-n, --nodelist {[nodelist]}] where the nodelist is a cluster description as
           described above.
         \item[-c --components {[component list]}].  The component list is a blank-delimited
-          list of components includeing {\em or, rm, sm, pm, ws, broker} to start an
+          list of components including {\em or, rm, sm, pm, ws, broker} to start an
           individual component, or {\em all} to start all of the components.  NOTE: It is
           usually an error to start any of these components more than once.  However 
           {\em start\_sim} allows it, to permit error injection.
@@ -196,12 +195,12 @@ agentn.79 = 6
         surest way to stop a simulated DUCC cluster.  In order for this to work,
         be sure to include the names of all physical nodes used in the simulated cluster
         in the DUCC configuration file {\em \duccruntime/resources/ducc.nodes.}  It
-        is descripted in the \hyperref[subsec:admin.check-ducc]{administration section} of the book.
+        is described in the \hyperref[subsec:admin.check-ducc]{administration section} of the book.
 
       \item {\em stop\_sim} With no arguments, this attempts to stop all the simulated
         agents and the management daemons using {\em kill -INT}.  It is possible to
-        stop individual agents or managemnt nodes by specifing their component IDs.
-        The kill signeals {\em -KILL, -STOP} and {\em -CONT} are all supported.  This
+        stop individual agents or management nodes by specifying their component IDs.
+        The kill signals {\em -KILL, -STOP} and {\em -CONT} are all supported.  This
         allows error injection as well as a more orderly shutdown than 
         {\em check\_ducc -k}.
     \end{enumerate}
@@ -229,10 +228,10 @@ agentn.79 = 6
     \begin{description}
       \item[-c, --component {[component name]}] where the name is one of {\em
         rm, sm, pm, or. ws,}.  {\em Kill -INT} is used to enable orderly shutdown
-      unless overriden with -k, -p, or -r as described below.
+      unless overridden with -k, -p, or -r as described below.
       \item[-i, --instance {[instance-id]}] where the instance-id is one of the
         agent ids in ``sim.pids''. {\em Kill -INT} is used to enable orderly shutdown
-      unless overriden with -k, -p, or -r as described below.
+      unless overridden with -k, -p, or -r as described below.
       \item[-k, --kill] Use {\em kill -9} to kill the process.
       \item[-p, --pause] Use {\em kill -STOP} to kill the process.
       \item[-r, --resume] Use {\em kill -CONT} to kill the process.
@@ -243,7 +242,7 @@ agentn.79 = 6
     \subsection{Overview}
      ``Real'' jobs are highly memory and CPU intensive.  For testing and simulation
      purposes, the jobs need not use anywhere close to their declared memory, and
-     need to consume any CPU at all.  The FixedSleepAE is a UIMA analytic that
+     need not consume any CPU at all.  The FixedSleepAE is a UIMA analytic that
      is given a time, in milliseconds, and all it does is sleep for that period
      of time and then exit.  By running many of these in a simulated cluster
      it is possible to get all the DUCC administrative processes to behave 
@@ -258,8 +257,8 @@ agentn.79 = 6
 
      The test driver is given a control file with the names of all the jobs to be
      submitted in the current run, and the elapsed time to wait between submission
-     of each job. Each jobname corresponds to a file that is not an actual
-     DUCC specifiction, but rather the description of a DUCC specification.  Each
+     of each job. Each job name corresponds to a file that is not an actual
+     DUCC specification, but rather the description of a DUCC specification.  Each
      description is a simple Java properties file.
 
      To submit a job, the test driver reads the next job description file
@@ -276,16 +275,16 @@ agentn.79 = 6
      \subsection{Job meta-descriptors}
      For each simulated job in a run, a meta-descriptor must be constructed.  These may be
      constructed ``by hand'', or via local scripting, for example from log analysis.  (The
-     packaged meta-desciptors are generated from logs of actual workloads.)
+     packaged meta-descriptors are generated from logs of actual workloads.)
 
      A meta-descriptor must contain the following properties:
      \begin{description}
-       \item[tod] This specifies a virual ``time of day of submission'', starting from time 0, specified
+       \item[tod] This specifies a virtual ``time of day of submission'', starting from time 0, specified
          in units of milliseconds, when the job is to be submitted.  During job generation, this may
          be used to enforce precise timing of submission of the jobs.
        \item[elapsed] This is a space-delimited set of numbers.  Each number represents the elapsed time,
          in milliseconds, for a single work item.  There must be one time for each work item.  
-         These numbers are placed into CASs by the job's Job Driver and deliverd to each Job Process.
+         These numbers are placed into CASs by the job's Job Driver and delivered to each Job Process.
          For example,
          if this job is to consist of 5 work items of 1, 2, 3, 4 and 5 seconds each, specify
 \begin{verbatim}
@@ -317,9 +316,9 @@ machines = 11
        All the job meta-descriptors for a run must be placed into a single directory.
 
 
-     \subsection{Run Descriptors}
+     \subsection{Prepare Descriptors}
      \label{subsec:simulation.run-description}
-     A {\em run descriptor}, or alternatively, {\em prepare descriptor} is also a
+     A  {\em prepare descriptor} is also a
      standard Java properties file.  This defines where the set of meta descriptors resides,
      where to place the modified meta-files, how to assign scheduling classes to the
      jobs, how to apportion memory sizes, how to apportion services, how long the total
@@ -347,7 +346,7 @@ machines = 11
          above.  The value is a weight, used to affect the distribution of memory sizes
          among the jobs.
        \item[job.services] This is a blank-delimited list of a service id, where the id
-         is one of the services specified in the {\em services.boot} conrol file.
+         is one of the services specified in the {\em services.boot} control file.
        \item[job.services.{[id]}] Here {\em id} is one of the ids specified in the
          job.services line above.  The value is a weight, used to affect the distribution
          of services among the jobs.
@@ -405,16 +404,15 @@ compression                   = 96      
      
      \subsection{Services}
      \label{subsec:simulation.services}
-     It is possible to run the FixedSleeAE as a UIMA-AS service, with each job
+     It is possible to run the FixedSleepAE as a UIMA-AS service, with each job
      specifying a dependency on the service, and the indicated service doing the
-     actual execution for the job (sleeping).
+     actual sleeping on behalf of the job.
 
-     Four variants on services are supported:
+     These variants on services are supported:
      \begin{enumerate}
-       \item Submitted services,
        \item Registered services, started by reference,
        \item Registered services, started by the simulator,
-       \item And standalone services, started independently of DUCC and discovered
+       \item Standalone services, started independently of DUCC and discovered
          by reference from a job.
      \end{enumerate}
 
@@ -423,13 +421,11 @@ compression                   = 96      
 
      Properties required in the service boot file include:
      \begin{description}
-       \item[submit] This specifies submitted services.  The value is a blank delimited
-         list of pseudo IDs for the submitted services.  
        \item[register] This specifies registered services.  The value is a blank delimited
          list of pseudo IDs for the registered services.
-       \item[stert] This specifies which of the registered services to automatically 
+       \item[start] This specifies which of the registered services to automatically 
          start.  The value is some subset of the pseudo IDS specified under {\em register}
-       \item[standalone] This specifes the pseudo IDs of the standalone (non-DUCC) services.
+       \item[standalone] This specifies the pseudo IDs of the standalone (non-DUCC) services.
        \item[instances\_{[id]}] Here {\em id} is one of the IDs specified for {\em submit,
            register,} or {\em standalone}.  The value is the number of instances of that
          specific service to set up.
@@ -441,32 +437,27 @@ compression                   = 96      
      the simulated runs to activate and use multiple independent services.  Because the
      endpoints are in the various UIMA XML service descriptors, it is necessary to use
      exactly these IDs when generating a test run.  Thus, the only valid pseudo-ids
-     for service confuration are {\em 0, 1, 2, 3, 4, 5, 6, 7, 8, 9}.
+     for service configuration are {\em 0, 1, 2, 3, 4, 5, 6, 7, 8, 9}.
 
      These {\em service ids} are used on the job configuration file to establish a
-     weghted distribution of service use among the jobs.
+     weighted distribution of service use among the jobs.
 
      Here is a sample service configuration file:
 \begin{verbatim}
-# submit these services, 2 instances each
-submit  0 1
+# register these services, 2 instances each
+register 0 1 2 3
 instances_0 2
 instances_1 2
-
-# register these services, 2 instances each
-register 2 3 4 5
 instances_2 2
 instances_3 2
-instances_4 2
-instances_5 2
 
 # start these registered services
 start 2 3 
 
 # start 2 standalone services
-standalone 6 7 
-instances_6 1
-instances_7 1
+standalone 4 5 
+instances_4 1
+instances_5 1
 \end{verbatim}
 
      \subsection{Generating a Job Set}
@@ -477,10 +468,10 @@ instances_7 1
 \begin{verbatim}
 prepare [config-file]
 \end{verbatim}
-     where {\em configile} is the \hyperref[subsec:simulation.run-description]{run description} file
+     where {\em config file} is the \hyperref[subsec:simulation.run-description]{run description} file
      described above.
 
-     This script reads the meta-spcification in the {\em jobs.in} directive of the
+     This script reads the meta-specification in the {\em jobs.in} directive of the
      config-file, generates a set of meta-specification files into the {\em jobs.out}
      directory, and creates a control file, {\em job.ctl}.  The {\em job.ctl} file is used
      by the simulation driver to submit all the jobs.
@@ -516,17 +507,20 @@ prepare [config-file]
         as configured, pauses a bit to let them start up, and generated every job with a
         dependency on one of the services.
       \item[-i time-in-sec] If specified, this forces each AE to spend a minimum of the indicated time
-        in it's initialization method (also a sleep). If not specified, the defaultis
-        10 seconds. The actual time is controled by the {\em -r} (range) option.
+        in it's initialization method (also a sleep). If not specified, the default is
+        10 seconds. The actual time is controlled by the {\em -r} (range) option.
       \item[-r time-in-sec] This specifies the top range for initialization.  The service
         will spend the time specified in {\em -i}, PLUS a random value from 1 to
         the time specified in {\em -r} in its initialization phase.
       \item[--IB] The Job Process will leak memory in it's initialization phase until it is killed, hopefully by
         DUCC, but possibly by the operating system.  {\em Use with care.}.
-      \item[--PB] The job Process will leak memory in it's processing phase until it is killed, hoefully 
+      \item[--PB] The job Process will leak memory in it's processing phase until it is killed, hopefully 
         by DUCC, but possibly by the operation system. {\em Use with care.}
       \item[-m size-in-gb] Memory override.  Use this value for all jobs, overriding the value
         in the generated meta-specification file.
+      \item[-n max-Number-of-processes] Max machine override.  If specified, this overrides the configured process max
+        from the job control file.  Specify the max as $0$ and no maximum will be submitted with the job,
+        causing the scheduler to try to allocated the largest possible number of processes to the job.
       \item[-p time-in-seconds] Generated the {\em time-in-seconds} as the maximum processing time
         for a work item.  DUCC will terminate any work item that exceeds this time.
       \item[-w, --watch] Submit every job with the {\em wait\_for\_completion} flag. This runs the
@@ -545,7 +539,7 @@ prepare [config-file]
 
     
 \section{Pre-Packaged Tests}
-    Three test suites are provided using the mechanisms described in the previoius section:
+    Three test suites are provided using the mechanisms described in the previous section:
     \begin{itemize}
       \item A 15-minute run comprising approximately 30 jobs.  This includes configuration for
         single-class submission, mixed class submission, and one configured to maximize
@@ -553,11 +547,13 @@ prepare [config-file]
       \item A 30-minute run comprising approximately 33 jobs. This includes a single
         configuration.
       \item a 24-hour run comprising approximately 260 jobs.  This also includes configurations
-        for single-class submission, mixed classes, and fragmentation.
+        for single-class submission, mixed classes, and fragmentation. {\em Note: this run has
+          been reconfigured to run in 12 hours, and has been successfully been configured
+          to complete in 6 hours.  This can create a significant load on the DUCC processes.}
     \end{itemize}
 
      The configurations are found in the \duccruntime/examples/systemtest directory
-     and are in subdirectories called, 
+     and are in sub directories called, 
      \begin{itemize}
        \item mega-15-min
        \item mega-30-min
@@ -575,7 +571,7 @@ prepare [config-file]
        \item Update your {\em \duccruntime/resources/ducc.mpdes} so that all the real node names specified
          int the simulated node file are included.
        \item Update your {\em \duccruntime/resources/ducc.properties} so the 
-         {\em ducc.head} is specifed as the {\em real, physical} machine where you will
+         {\em ducc.head} is specified as the {\em real, physical} machine where you will
          start the simulated cluster.
        \item Be sure the {\em job driver} nodepool, if configured in
          {\em \duccruntime/resources/ducc.classes}, specifies the name of one of the