You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uima.apache.org by cw...@apache.org on 2013/01/02 20:12:11 UTC
svn commit: r1427917 [8/8] - in
/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook: ./ images/
images/ducc-overview/ images/job-manager/ part-admin/ part-admin/admin/
part-introduction/ part-user/ part-user/cli/ unused/
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/services_unregister.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/submit.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/submit.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/submit.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/submit.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,1040 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+ <!-- ************************************ SUBMIT CLI ********************************** -->
+ <section id="ducc.cli.submit">
+ <title>ducc_submit</title>
+
+ <para>
+ <emphasis>The source for this section is ducc_ducbook/documents/part-user/cli/submit.xml</emphasis>
+ </para>
+
+
+ <para>
+ <emphasis>Description:</emphasis>
+ <variablelist>
+ <varlistentry>
+ <term></term>
+ <listitem>
+
+ <para>
+ The <emphasis>submit</emphasis> CLI is used to submit work for execution by DUCC. DUCC assigns a unique id
+ to the job and schedules it for execution. The submitter may optionally request that the progress of the
+ job is monitored, in which case the state of the job as it progresses through its lifetime is printed on
+ the console.
+ </para>
+
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ <emphasis>Usage:</emphasis>
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+
+ <variablelist>
+ <varlistentry>
+ <term><emphasis role="bold">Script wrapper</emphasis></term>
+ <listitem><para><emphasis>$DUCC_HOME</emphasis>/bin/ducc_submit</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><emphasis role="bold">Executble Jar</emphasis></term>
+ <listitem><para>java -jar <emphasis>$DUCC_HOME</emphasis>/lib/ducc-submit.jar</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><emphasis role="bold">Java main</emphasis></term>
+ <listitem><para>org.apache.uima.ducc.cli.DuccJobSubmit</para></listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <para>
+ If no options are given, help text is presented.
+ </para>
+
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ <emphasis>Options:</emphasis>
+ <variablelist>
+
+ <varlistentry>
+ <term></term>
+ <listitem>
+ <para>
+ <emphasis role="bold">--cancel_job_on_interrupt</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ If the job is started
+ with <emphasis><link linkend="ducc.cli.submit.wait_for_completion">--wait_for_completion</link></emphasis>,
+ this option causes the job to be canceled with Ctrl-C.
+ If <emphasis>--cancel_job_on_interrupt</emphasis> is not specified, the job
+ monitor will be terminated but the job will continue to run.
+ </para>
+ <para>
+ If <emphasis>--wait_for_completin</emphasis> is not specified this option is ignored.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--debug</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ Enable debugging messages. This is primarily for debugging DUCC itself.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--description [text]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ The text is any string used to describe the job. It is displayed in the Web Server.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_classpath [classpath]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This is the classpath for the Job Driver, necessary for DUCC to find the
+ Collection Reader.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_descriptor_CR [descriptor.xml]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This is the XML descriptor for the Collection Reader. It is searched for as a resource
+ as described above.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_descriptor_CR_overrides [list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This is the Job Driver collection reader configuration overrides. They are specified
+ as name/value pairs in a comma-delimeted list. For example:
+ <screen>--driver_descriptor_CR_overrides name1=value1,name2=value2...</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_environment</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies environment parameters for the Job Driver. If present, they are added
+ to the Job Driver's environment as the process is spawned. It must be a quoted,
+ blank-delimeted lsit of name-value pairs. For example:
+ <screen>"TERM=xterm DISPLAY=:1.0"</screen>
+ </para>
+ <note>
+ <para>
+ On Secure Linux systems, the environemnt variable LD_LIBRARY_PATH may not be passed
+ to the user's program. If it is necessary to pass LD_LIBRARY_PATH to the JP or
+ JD processes, it must be specified as DUCC_LD_LIBRARY_PATH. Ducc (<emphasis>securely</emphasis>)
+ passes this as LD_LIBRARY_PATH, <emphasis>after</emphasis> the JP or JD has assumed
+ the user's identity. For example:
+ <screen>"--process_environment TERM=xterm DISPLAY=:1.0 DUCC_LD_LIBRARY_PATH=/my/own/lib.so"</screen>
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_jvm_args</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifes extra JVM arguments to be provided to the Job Driver process. It is a blank-delimeted list
+ of strings. Example:<screen>--driver_jvm_args -Xmx100M -Xms50M</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--driver_memory_size [size-in-GB]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the size of memory for the Job Driver, in GB. Example:<screen>--driver_memory_size 16</screen>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--help</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ Prints the usage text to the console.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--jvm [path-to-java]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ States the JVM to use. If not specified, the same JVM used by the Agents is used.
+ Example:<screen>--jvm /share/jdk1.6/bin/java</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--log_directory [path-to-log directory]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the path to the directory for the user logs. If not specified,
+ the default is the user's home directory. Example:<screen>--log_directory /home/bob</screen>.
+ Within this directory DUCC creates a subdirectory for each job, using the
+ numerical ID of the job. The format of the generated log file names is
+ descripbed in <xref linkend="ducc.user.logs" />.
+ </para>
+ <note>
+ <para>
+ Note that <emphasis>--log_directory</emphasis> specifies only the path to
+ a directory where logs are to be stored. In order to manage multiple processes
+ running in multiple machines DUCC, sub-directory and file names are generated
+ by DUCC and may not be directly specified.
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_classpath [ClASSPATH]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the Java CLASSPATH to use in each Job Process (JP) and must be specified. Example:
+ <screen>--process_classpath a.jar:b.jar</screen>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_DD [DD descriptor]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies a UIMA Deployment Descriptor for the job processes for DD-style
+ jobs. This is mutually exclusive with <emphasis>--process_descriptor_AE,
+ --process_descriptor_CM, and --process_descriptor_CC.</emphasis> This descriptor
+ is a resource that is searched for in the CLASSPATH and data path as described
+ in the <emphasis><link linkend="ducc.cli.submit.notes">notes</link></emphasis>. For
+ example:<screen>--process_DD /home/billy/resource/DD_foo.xml</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_deployments_max [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the maximum nunber of Job Processes to deploy at any given time. If not
+ specified, DUCC will attempt to provide the largest number of processes, within the
+ constraints of <link linkend="ducc.glossary.fair-share">fair_share </link> scheduling and the number of
+ pending <link linkend="ducc.glossary.fair-share">work items</link>
+ still to be done in the job.
+ <screen>--process_deployments_max 66</screen>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_AE [descriptor]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies Analysis Engine descriptor to be deployed in the Job Processes.
+ This descriptor is a resource that is searched for in the CLASSPATH and data
+ path as described in the <emphasis>notes</emphasis>. It is mutually exclusive
+ with <emphasis>--process_DD</emphasis>
+ For
+ example:<screen>--process_AE /home/billy/resource/AE_foo.xml</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_AE_overrides [list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies AE overrides. It is a comma-delimeted list of
+ name/value pairs. Example:<screen>--process_descriptor_AE_Overrides name1=value1,name2=value2</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_CC [descriptor]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the CAS Consumer descriptor to be deployed in the Job Processes.
+ This descriptor is a resource that is searched for in the CLASSPATH and data
+ path as described in the <emphasis>notes</emphasis>. It is mutually exclusive
+ with <emphasis>--process_DD</emphasis>
+ For
+ example:<screen>--process_descriptor_CC /home/billy/resourceCCE_foo.xml</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_CC_overrides [list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies CC overrides. It is a comma-delimeted list of
+ name/value pairs. Example:<screen>--process_descriptor_CC_overrides name1=value1,name2=value2</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_CM [descriptor]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the CAS Multiplier descriptor to be deployed in the Job Processes.
+ This descriptor is a resource that is searched for in the CLASSPATH and data
+ path as described in the <emphasis>notes</emphasis>. It is mutually exclusive
+ with <emphasis>--process_DD</emphasis>
+ For
+ example:<screen>--process_descriptor_CM /home/billy/resource/CM_foo.xml</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_descriptor_CM_overrides [list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies CM overrides. It is a comma-delimeted list of
+ name/value pairs. Example:<screen>--process_descriptor_CM_overrides name1=value1,name2=value2</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_environment [environment]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies environment parameters for the Job Processes. If present, they are added
+ to the Job Process environment as the process is spawned. It must be a quoted,
+ blank-delimeted lsit of name-value pairs. For example:
+ <screen>"--process_environment TERM=xterm DISPLAY=:1.0"</screen>
+ </para>
+ <note>
+ <para>
+ On Secure Linux systems, the environemnt variable LD_LIBRARY_PATH may not be passed
+ to the user's program. If it is necessary to pass LD_LIBRARY_PATH to the JP or
+ JD processes, it must be specified as DUCC_LD_LIBRARY_PATH. Ducc (<emphasis>securely</emphasis>)
+ passes this as LD_LIBRARY_PATH, <emphasis>after</emphasis> the JP or JD has assumed
+ the user's identity. For example:
+ <screen>"--process_environment TERM=xterm DISPLAY=:1.0 DUCC_LD_LIBRARY_PATH=/my/own/lib.so"</screen>
+ </para>
+ </note>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_failures_limit [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the maximum number of individual Job Process (JP) failures that are
+ to be tolerated before killing the job. The default is 15. If this limit is
+ exceeded over the lifetime of a job DUCC terminates the entire job.
+ <screen>"--process_failures_limit 23"</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_get_meta_time_max [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ When a job is started the Job Driver issus a single "get-meta" requests to the
+ (DUCC-generated) endpoint of the JP processes for the job to insure that at least one UIMA-AS
+ server processes for the job have started.
+ This parameter specifies the time in
+ seconds to wait for a response. If the request times out the Job Driver assumes
+ that no UIMA-AS service for the job was able to start and it terminates the job. If not specified,
+ the timeout is 2 minutes. Example:
+ <screen>"--process_get_meta_time_max 10"</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_initialization_failures_cap [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the maximum number of independent Job Process initialization
+ failures (i.e. System.exit(), kill-15...) before the number of Job Processes is
+ capped at the number in state Running currently. The default is 99.
+ Example:<screen>--process_initialization_failures_cap 62</screen>
+ </para>
+ <para>
+ Note that the job is NOT killed if there are processes that have passed initialization and
+ are running. If this limit is reached, the only action is to not start new processes
+ for the job.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_jvm_args [list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies additinal arguments to be passed to the Job Process JVM.
+ Example:<screen>--process_jvm_args -Xmx400M -Xms100M</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_memory_size [size]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the maximum amount of RAM in GB to be allocated to each Job Process. This
+ value is used by the Resource Manager to allocate resources. if this amount is
+ exceeded by a Job Process the Agent terminates the process with a ShareSizeExceeded
+ message.
+ Example:<screen>--process_memory_size 33</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_per_item_time_max [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the maximum time in minutes that the Job Driver will wait for a
+ Job Processes to process a CAS. If a timeout occurs the process is terminated
+ and the CAS marked in error (not retried). If not specified, the default
+ is 1 minute.
+ Example:<screen>--process_per_item_time_max 60</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--process_thread_count [integer]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the number of threads per process to be deployed. It is used by the
+ Resource Manager to determine how many processes are needed, by the Agent to
+ determine howmany threads to spawn, and by the Job Driver to determine how
+ many CASs to dispatch. If not specified, the default is 4.
+ Example:<screen>--process_thread_count 7</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--scheduling_class [classname]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the name of the scheuling class the RM will use to determine
+ the resource allocation for each process. The names of the classes are
+ installation dependent. If not specified, the default is taken
+ from the global DUCC configuration <link linkend="ducc.properties">ducc.properties</link>.
+ Example:<screen>--schedling_class normal</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--service_dependency[list]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies a comma-delimeted list of services the job processes are dependent upon.
+ Each endpoint must be of the form UIMA-AS:<emphasis>endpoint</emphasis>:<emphasis>broker_url</emphasis>
+ where <emphasis>endpoint</emphasis> is the UIMA-AS service endpoint and <emphasis>broker_url</emphasis>
+ is the ActiveMQ broker URL.
+ </para>
+ <para>
+ In the example are two dependencies, one with endpoint RandomSleepAE and broker
+ tcp:bluej682:61616, and the other with endpoint OtherEp and broker URL
+ tcp:bluej123:123. Example:
+ <literallayout>
+ --service_dependency UIMA-AS:RandomSleepAE:tcp:bluej682:61616, \
+ UIMA-AS:OtherEp:tcp:bluej123:123
+ </literallayout>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--specifiecaiton [file]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ All the parameters used to submit a job may be placed in a standard Java properties file. This file
+ may then be used to submit the job (rather than providing all the parameters directory to submit).
+ </para>
+ <para>
+ For example, <screen>ducc_submit --specification job.props</screen> where the <emphasis>job.props</emphasis>
+ contains:
+ <screen>
+working_directory=/Users/challngr/projects/ducc/ducc_test/test/bin
+process_get_meta_time_max=5
+process_failures_limit=20
+driver_descriptor_CR=org.apache.uima.ducc.test.randomsleep.FixedSleepCR
+driver_environment=DUCC_LD_LIBRARY_PATH=/a/other/bogus/path
+process_environment=AE_INIT_TIME=10000 DUCC_LD_LIBRARY_PATH=/a/bogus/path
+driver_classpath=/home/bob/duccapps/ducky_driver.jar
+log_directory=/Users/challngr/ducc/logs/
+process_thread_count=1
+driver_descriptor_CR_overrides=jobfile:../simple/jobs/1.job,compression:10
+process_initialization_failures_cap=99
+process_per_item_time_max=60
+driver_jvm_args=-Xmx500M
+process_descriptor_AE=org.apache.uima.ducc.test.randomsleep.FixedSleepAE
+process_classpath=/home/bob/duccapps/ducky_process.jar
+description=../simple/jobs/1.job[AE]
+process_jvm_args=-Xmx100M -DdefaultBrokerURL=tcp://localhost:61616
+scheduling_class=normal
+process_memory_size=15
+ </screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--timestamp</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ If specified, messages from the submit process are timestamped. This is intended primarily for use
+ with a monitor with <emphasis><link linkend="ducc.cli.submit.wait_for_completion"> --wait_for_completion</link></emphasis>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="ducc.cli.submit.wait_for_completion">
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--wait_for_completion</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ If specified, the submit command does not return control to the consoke immediately, and instead
+ monitors the DUCC state traffic and prints information about the job as it progresses.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--working_directory</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ This specifies the working directory to be set by the Job Driver and Job Process processes. If not
+ specified, the current directory is used. Example:<screen>--working_directory /Users/challngr/projects/ducc/ducc_test/bin</screen>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ <para id="ducc.cli.submit.notes">
+ <emphasis>Notes:</emphasis>
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+
+ <para>
+ When searching for UIMA XML resource files such as descriptors, DUCC searches both the classpath and the
+ data path according to the following rules:
+ <orderedlist>
+ <listitem>
+ <para>
+ If the resource ends in .xml it is assumed the resource is a file and
+ the path is either an absolute path or a path relative to the specified
+ working directory. If the file is not found the search exits and the job
+ is terminated.
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ If the resource does not end in .xml, DUCC creates a path by replacing the
+ "." separators with "/" and appending ".xml". It then searches two places:
+ <orderedlist>
+ <listitem>
+ <para>
+ The user's CLASSPATH as a file (that is, not in a jar), and
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ In the jar files provided in the user's CLASSPATH.
+ </para>
+ </listitem>
+ </orderedlist>
+ </para>
+ <para>
+ If the resource is found in either place the search is successful. Otherwise the
+ search fails and the job is terminated.
+ </para>
+ </listitem>
+ </orderedlist>
+ </para>
+
+ <para>
+ The resource search-order rules apply to all of the following submit parameters:
+ <itemizedlist>
+ <listitem>
+ <para>
+ --driver_descriptor_CR
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ --process_descriptor_AE
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ --process_descriptor_CC
+ </para>
+ </listitem>
+
+ <listitem>
+ <para>
+ --process_descriptor_CM
+ </para>
+ </listitem>
+ </itemizedlist>
+
+ </para>
+
+
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ </section>
+
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/submit.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/unreserve.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/unreserve.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/unreserve.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/unreserve.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,127 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+ <!-- ************************************ Unreserve CLI ********************************** -->
+ <section>
+ <title>ducc_unreserve</title>
+
+ <para>
+ <emphasis>The source for this section is ducc_ducbook/documents/part-user/cli/unreserve.xml</emphasis>
+ </para>
+
+ <para>
+ <emphasis>Description:</emphasis>
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+
+ <para>
+ The <emphasis>unreserve</emphasis> CLI is used to release reserved resources.
+ </para>
+
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ <emphasis>Usage:</emphasis>
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+
+ <variablelist>
+ <varlistentry>
+ <term><emphasis role="bold">Script wrapper</emphasis></term>
+ <listitem><para><emphasis>$DUCC_HOME</emphasis>/bin/ducc_unreserve</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><emphasis role="bold">Executble Jar</emphasis></term>
+ <listitem><para>java -jar <emphasis>$DUCC_HOME</emphasis>/lib/ducc-unreserve.jar</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term><emphasis role="bold">Java main</emphasis></term>
+ <listitem><para>org.apache.uima.ducc.cli.DuccReservationCancel</para></listitem>
+ </varlistentry>
+
+ </variablelist>
+
+ <para>
+ If no options are given, help text is presented.
+ </para>
+
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+
+ <para>
+ <emphasis>Options:</emphasis>
+ <variablelist>
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--id [jobid]</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ The ID is the reservation ID returned by the job submission.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ <emphasis role="bold">--help</emphasis>
+
+ <variablelist>
+ <varlistentry>
+ <term />
+ <listitem>
+ <para>
+ Prints the usage text to the console.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+
+ </variablelist>
+ </para>
+
+ </section>
+
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/cli/unreserve.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/user-classpath.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/user-classpath.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/user-classpath.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/user-classpath.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<simplelist>
+ <member>$DUCC_HOME/lib/ducc-cli.jar</member>
+ <member>$DUCC_HOME/lib/ducc-common.jar</member>
+ <member>$DUCC_HOME/lib/apache-activemq-5.5.0/activemq-all-5.5.0.jar</member>
+ <member>$DUCC_HOME/lib/apache-commons-cli-1.2/commons-cli-1.2.jar</member>
+ <member>$DUCC_HOME/lib/apache-camel-2.7.1/*</member>
+ <member>$DUCC_HOME/lib/http-client/*</member>
+ <member>$DUCC_HOME/lib/springframework-3.0.5/*</member>
+ <member>$DUCC_HOME/uima/*</member>
+ <member>$DUCC_HOME/resources</member>
+</simplelist>
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/part-user/user-classpath.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-agent.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-agent.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-agent.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-agent.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<chapter id="ducc.agent.ov">
+<title>Overview - Node Agent</title>
+ <para>UIMA Asynchronous Scaleout (AS) is a set of capabilities supported in the UIMA Framework for achieving
+ scaleout that is more general than the approaches provided for in the Collection Processing Manager (CPM). AS is a
+ second generation design, replacing the CPM and Vinci Services. The CPM and Vinci are still available and are not
+ being deprecated, but new designs are encouraged to use AS for scalability, and current designs reaching
+ limitations may want to move to AS.</para>
+ <para>AS is integrated with the flow controller architecture, and can be applied to both primitive and aggregate
+ analysis engines. </para>
+</chapter>
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-agent.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-ducc-overview.xml.sav
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-ducc-overview.xml.sav?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-ducc-overview.xml.sav (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-ducc-overview.xml.sav Wed Jan 2 19:12:10 2013
@@ -0,0 +1,282 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<chapter id="ducc.ducc.ov">
+<title>Overview - Distributed UIMA Computing Cluster </title>
+ <para>What is DUCC?</para>
+
+<section id="ugr.async.ov.terminology">
+ <title>Terminology</title>
+ <para>Terms used in describing DUCC capabilities include: </para>
+ <variablelist>
+ <varlistentry>
+ <term> <emphasis role="bold">DUCC</emphasis></term>
+ <listitem>
+ <para>Distributed UIMA Computing Cluster</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Job Manager</emphasis></term>
+ <listitem>
+ <para>What is Job Manager?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Resource Manager</emphasis></term>
+ <listitem>
+ <para>What is Resource Manager?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Process Manager</emphasis></term>
+ <listitem>
+ <para>What is Process Manager?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Service Manager</emphasis></term>
+ <listitem>
+ <para>What is Service Manager?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Agent</emphasis></term>
+ <listitem>
+ <para>What is Agent?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">View Manager</emphasis></term>
+ <listitem>
+ <para>What is View Manager?</para></listitem>
+ </varlistentry>
+
+ <varlistentry>
+ <term> <emphasis role="bold">Fair Share Algorithm</emphasis></term>
+ <listitem>
+ <para>What is Share Fair Algorithm?</para></listitem>
+ </varlistentry>
+
+ </variablelist>
+</section>
+
+ <section id="ugr.async.ov.as_vs_cpm">
+ <title>AS versus CPM</title>
+ <para>It is useful to compare and contrast the approaches and capabilities of AS and CPM.</para>
+ <figure id="ugr.async.ov.fig.wrapAE">
+ <title>AS Primitive Wrapper</title>
+ <mediaobject>
+ <imageobject role="html">
+ <imagedata width="279px" format="PNG"
+ fileref="images/ducc-overview/wrapAE.png"></imagedata>
+ </imageobject>
+ <imageobject role="fo">
+ <imagedata width="2.7in" format="PNG"
+ fileref="images/ducc-overview/wrapAE.png"></imagedata>
+ </imageobject>
+ <textobject> <phrase>AS Primitive Wrapper</phrase></textobject></mediaobject></figure>
+ <informaltable pgwide="1" colsep="1" frame="all" rowsep="1">
+ <tgroup cols="3">
+ <colspec colname="c1" colwidth="1*"></colspec>
+ <colspec colname="c2" colwidth="2.5*"></colspec>
+ <colspec colname="c3" colwidth="2.5*"></colspec>
+ <thead>
+ <row>
+ <entry align="center"></entry>
+ <entry align="center">AS</entry>
+ <entry align="center">CPM</entry></row></thead>
+ <tbody valign="middle">
+ <row>
+ <entry>Putting components together</entry>
+ <entry>
+ <para>Provides a consistent, single, unified way to put components together, using
+ the base UIMA "aggregate" capability.</para>
+ </entry>
+ <entry>
+ <para> <emphasis role="bold">Two methods of putting components together</emphasis>
+ <orderedlist spacing="compact">
+ <listitem>
+ <para>CPE (Collection Processing Engine) descriptor, which has sections specifying a
+ Collection Reader, and a set of CAS Processors</para></listitem>
+ <listitem>
+ <para>Each CAS Processor can, as well, be an aggregate</para></listitem></orderedlist>
+ </para>
+ </entry></row>
+ <row>
+ <entry>Kinds of Aggregates</entry>
+ <entry>
+ <para>An aggregate can be run <emphasis role="bold">asynchronously</emphasis> using the AS
+ mechanism, with a queue in front of each delegate, or it can by run <emphasis role="bold">
+ synchronously</emphasis>.
+ </para>
+ <para>
+ When run asynchronously, <emphasis>all</emphasis> of the
+ delegates will have queues in front of them, and delegates which are AS Primitives can be individually scaled
+ out (replicated) as needed.
+ Also, multiple CASes can be in-process, at different steps in the pipeline, even without
+ replicating any components.</para>
+ </entry>
+ <entry>All aggregates are run synchronously. In an aggregate, only one component is running at a
+ time; there is only one CAS at a time being processed within the aggregate.</entry></row>
+ <row>
+ <entry>CAS flow</entry>
+ <entry>Any, including custom user-defined sequence using user-provided flow controller.
+ Parallel flows are supported.
+ </entry>
+ <entry>Fixed linear flow between CAS processors. A single CAS processor can be an aggregate, and within
+ the aggregate, can have any flow including custom user-defined sequence using user-provided flow
+ controller.</entry></row>
+ <row>
+ <entry>Threading</entry>
+ <entry>Each instance of a component runs in its own thread; the same thread used to call
+ <code>initialize()</code> for a particular instance of a component
+ is used when calling <code>process()</code>.</entry>
+ <entry>One thread for the collection reader, one for the CAS Consumers, "n" threads for the
+ main pipeline, with no guarantees that the same thread for the <code>initialize()</code> call
+ is used for the <code>process()</code> call.</entry></row>
+ <!--row>
+ <entry>Load Balancing</entry>
+ <entry>Queue in front of each (set of replicated) component(s); when component is available, it
+ "pulls" requests from the queue</entry>
+ <entry>One master queue; entries are "pushed" to components; limited load balancing capability.</entry>
+ </row-->
+ <row>
+ <entry>Delegate deployment</entry>
+ <entry>Co-located or remote.</entry>
+ <entry>Co-located or remote.</entry></row>
+ <!--row>
+ <entry>Remoting specifications</entry>
+ <entry>
+ <formalpara><title><emphasis role="bold">AS:</emphasis></title>
+ <para>XML deployment specification is used to specify
+ where each delegate is deployed, and protocol to use to connect to it.
+ </para>
+ </formalpara>
+ <formalpara><title><emphasis role="bold">Service Client Descriptor:</emphasis></title>
+ <para>Connect using Vinci or SOAP protocols, specified using
+ service client descriptor</para></formalpara>
+ </entry>
+
+ <entry>
+ <formalpara><title><emphasis role="bold">CPE:</emphasis></title>
+ <para>Limited to Vinci protocol, uses Vinci service client descriptor to connect to
+ a remote Analysis Engine service</para></formalpara>
+
+ <formalpara><title><emphasis role="bold">Service Client Descriptor:</emphasis></title>
+ <para>Connect using Vinci or SOAP protocols, specified using
+ service client descriptor</para></formalpara>
+ </entry>
+ <!-
+ <para> Both the CPE and aggregate methods of putting components together
+ support remoting. In addition, the CPE supports a limited form of life-cycle
+ management, for CAS processors running in the same Host machine, but in
+ different processes. The CPE specified remoting is limited to using the Vinci
+ communication protocol; an aggregate can use additional protocols,
+ such as SOAP.</para>
+
+ <para> Aggregates may specify remoting of delegates by having the delegate
+ descriptor be a service client descriptor for a remote client. Multiple
+ protocols are supported. The UIMA
+ framework handles connecting to the service, but does no life-cycle or error
+ recovery. </para>
+ </entry> - >
+ </row-->
+ <row>
+ <entry>Life cycle management</entry>
+ <entry>
+ <para>Scripts to launch services, launch Queue Brokers.</para>
+ </entry>
+ <entry>
+ <para>Scripts to launch services, start Vinci Name Service.</para>
+ <para>In addition, CPE "managed" configuration provides for automatic launching of
+ UIMA Vinci services in same machine, in different processes. </para>
+ </entry></row>
+ <row>
+ <entry>Error recovery</entry>
+ <entry>
+ <para>Similar capabilities as the CPM provides for CAS Processors, but at the finer granularity of
+ each AS component. The support includes customizable behavior overrides and extensions via user
+ code. </para>
+ </entry>
+ <entry>
+ <para>Error detection, thresholding, and recovery options at the granularity of CAS Processors
+ (which are CPM components, not delegates of aggregates), with some customizable callback
+ notifications</para>
+ </entry></row>
+ <row>
+ <entry>Firewall interactions</entry>
+ <entry>Enables deployment of AS services behind a firewall using a public broker. Enables deployment
+ of a public broker through single port, or using HTTP "tunneling".</entry>
+ <entry>When using Vinci protocol, requires opening a large number of ports for each deployed service.
+ SOAP connected services require one open port.</entry></row>
+ <row>
+ <entry>Monitoring and Tuning</entry>
+ <entry>
+ <para>JMX (Java Management Extensions) are enabled for recording many kinds of statistical
+ information, and can be used to monitor (and control) the operations of AS
+ configured systems. Statistics are provided and summarized from remote delegates, to aid in tuning
+ scaled-out deployments.</para>
+ </entry>
+ <entry>
+ <para>Some JMX information</para>
+ </entry></row>
+ <row>
+ <entry>Collection Reader</entry>
+ <entry>Supported for backwards compatibility. New programs should use the CAS Multiplier instead,
+ which is more general, or have the application pass in CASes to be processed. The compatibility
+ support wraps Collection Readers as Cas Multipliers. Note: this is supported and implemented in base UIMA.</entry>
+ <entry>Is always first element in linear CPE sequence chain</entry></row></tbody></tgroup>
+ </informaltable>
+ </section>
+
+
+ <section id="ugr.async.ov.design_goals">
+ <title>Design goals for Asynchronous Scaleout</title>
+ <para>The design goals for AS are:
+ <orderedlist spacing="compact">
+ <listitem>
+ <para>Increased flexibility and options for scaleout (versus CPM)</para>
+ <orderedlist spacing="compact">
+ <listitem>
+ <para>scale out parts independently of other parts, to appropriate degree</para></listitem>
+ <listitem>
+ <para>more options for protocols for remote connections, including some that don't require many
+ ports through firewalls</para></listitem>
+ <listitem>
+ <para>
+ support multiple CASes in process simultaneously within an aggregate pipeline</para></listitem>
+ </orderedlist></listitem>
+ <listitem>
+ <para>Build upon widely accepted Apache-licensed open source middleware</para></listitem>
+ <listitem>
+ <para>Simplification:
+ <orderedlist spacing="compact">
+ <listitem>
+ <para>Standardize on single approach to aggregate components</para></listitem>
+ <listitem>
+ <para>More uniform Error handling / recovery / monitoring for all AS managed components. </para>
+ </listitem>
+ <listitem><para>No changes to existing annotator code or descriptors. An additional deployment
+ descriptor is used to augment the conventional descriptors.</para></listitem>
+ </orderedlist> </para></listitem></orderedlist> </para>
+</section>
+
+
+
+</chapter>
\ No newline at end of file
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-orchestrator.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-orchestrator.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-orchestrator.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-orchestrator.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,90 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<chapter id="ducc.jm.ov">
+<title>Overview - Orchestrator</title>
+<section>
+ <title>Introduction</title>
+ <para>
+ The JM component supervises the lifecycle of UIMA applications across a distributed set of computing resources.
+ </para>
+ <para>
+ Users interact with the JM to submit, modify and cancel jobs.
+ The JM continually communicates with other DUCC components to schedule and dispatch work on the distributed cluster
+ of machines.
+ </para>
+ <para>
+ Typically, each job comprises two parts: CR and CM+AE+CC aggregate, with one CR providing CASes to many CM+AE+CC aggregates
+ distributed on the cluster.
+ </para>
+ <para>
+ Each aggregate is deployed as parallel UIMA-AS services which may increase or decrease in number over
+ the lifetime of the job, as prescribed by the RM.
+ </para>
+</section>
+<section>
+ <title>Job States</title>
+ <para>
+ Each job traverses a simple state machine from inception to conclusion.
+ </para>
+ <figure id="jm.ov.fig.States">
+ <title>Job States</title>
+ <mediaobject>
+ <imageobject role="html">
+ <imagedata width="700px" format="GIF"
+ fileref="images/job-manager/DUCC.JobsManager.JobStates.v2.gif"></imagedata>
+ </imageobject>
+ <imageobject role="fo">
+ <imagedata width="7.0in" format="GIF"
+ fileref="images/job-manager/DUCC.JobsManager.JobStates.v2.gif"></imagedata>
+ </imageobject>
+ <textobject> <phrase>Job States</phrase></textobject></mediaobject>
+ </figure>
+ <para>
+ Jobs enter the DUCC system upon submission and exit upon
+ the conclusion of processing the last work item or
+ the cancellation of the job by command or
+ by exceeding a configured error threshold.
+ </para>
+</section>
+<section>
+ <title>Process States</title>
+ <para>
+ One or more processes comprising UIMA/SA services are deployed during the lifetime of
+ a Job. Normally, each process goes through an initialization phase followed by the
+ processing of one or more work items.
+ </para>
+ <para>
+ Each UIMA/AS service traverses a simple state machine from inception to conclusion.
+ </para>
+ <figure id="jm.ov.fig.Processes">
+ <title>Job Processes States</title>
+ <mediaobject>
+ <imageobject role="html">
+ <imagedata width="700px" format="GIF"
+ fileref="images/job-manager/DUCC.JobsManager.ProcessStates.v2.gif"></imagedata>
+ </imageobject>
+ <imageobject role="fo">
+ <imagedata width="7.0in" format="GIF"
+ fileref="images/job-manager/DUCC.JobsManager.ProcessStates.v2.gif"></imagedata>
+ </imageobject>
+ <textobject> <phrase>Job States</phrase></textobject></mediaobject>
+ </figure>
+</section>
+</chapter>
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-orchestrator.xml
------------------------------------------------------------------------------
svn:eol-style = native
Added: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-process-manager.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-process-manager.xml?rev=1427917&view=auto
==============================================================================
--- uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-process-manager.xml (added)
+++ uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-process-manager.xml Wed Jan 2 19:12:10 2013
@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<chapter id="ducc.pm.ov">
+<title>Overview - Process Manager</title>
+ <para>UIMA Asynchronous Scaleout (AS) is a set of capabilities supported in the UIMA Framework for achieving
+ scaleout that is more general than the approaches provided for in the Collection Processing Manager (CPM). AS is a
+ second generation design, replacing the CPM and Vinci Services. The CPM and Vinci are still available and are not
+ being deprecated, but new designs are encouraged to use AS for scalability, and current designs reaching
+ limitations may want to move to AS.</para>
+ <para>AS is integrated with the flow controller architecture, and can be applied to both primitive and aggregate
+ analysis engines. </para>
+</chapter>
Propchange: uima/sandbox/uima-ducc/trunk/uima-ducc-ducbook/docbook/unused/chapter-process-manager.xml
------------------------------------------------------------------------------
svn:eol-style = native