You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@apex.apache.org by th...@apache.org on 2016/02/29 08:03:43 UTC
[08/46] incubator-apex-core git commit: SPOI-6656 #resolve Adding
development setup guide; fixing package docs
SPOI-6656 #resolve Adding development setup guide; fixing package docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/1cbcb0f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/1cbcb0f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/1cbcb0f4
Branch: refs/heads/APEXCORE-293
Commit: 1cbcb0f48b5556eb68f628e6c61ecf7001e16764
Parents: ac56f51
Author: sashadt <sa...@datatorrent.com>
Authored: Fri Oct 30 18:56:10 2015 -0700
Committer: Thomas Weise <th...@datatorrent.com>
Committed: Sun Feb 28 22:46:33 2016 -0800
----------------------------------------------------------------------
ApexPackage.md | 685 -----------------------------------------
AppConfPackage.md | 246 ---------------
apex_development_setup.md | 153 +++++++++
application_packages.md | 684 ++++++++++++++++++++++++++++++++++++++++
configuration_packages.md | 237 ++++++++++++++
5 files changed, 1074 insertions(+), 931 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/1cbcb0f4/ApexPackage.md
----------------------------------------------------------------------
diff --git a/ApexPackage.md b/ApexPackage.md
deleted file mode 100644
index 309b7ba..0000000
--- a/ApexPackage.md
+++ /dev/null
@@ -1,685 +0,0 @@
-#Apache Apex (incubating) Application Packages
-
-# Introduction
-
-An Apache Apex Application Package is a zip file that contains all the
-necessary files to launch an application in Apache Apex. It is the
-standard way for assembling and sharing an Apache Apex application.
-
-# Requirements
-
-You will need have the following installed:
-
-1. Apache Maven 3.0 or later (for assembling the App Package)
-2. Apache Apex 3.0.0 or later (for launching the App Package in your cluster)
-
-# Creating Your First Apex App Package
-
-You can create an Apex Application Package using your Linux command
-line, or using your favorite IDE.
-
-## Using Command Line
-
-First, change to the directory where you put your projects, and create
-an Apex application project using Maven by running the following
-command. Replace "com.example", "mydtapp" and "1.0-SNAPSHOT" with the
-appropriate values (make sure this is all on one line):
-
-```
- $ mvn archetype:generate
- -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito
- ries/releases
- -DarchetypeGroupId=com.datatorrent
- -DarchetypeArtifactId=apex-app-archetype -DarchetypeVersion=3.2.0
- -DgroupId=com.example -Dpackage=com.example.mydtapp -DartifactId=mydtapp
- -Dversion=1.0-SNAPSHOT
-```
-
-This creates a Maven project named "mydtapp". Open it with your favorite
-IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). In the project, there is a
-sample DAG that generates a number of tuples with a random number and
-prints out "hello world" and the random number in the tuples. The code
-that builds the DAG is in
-src/main/java/com/example/mydtapp/Application.java, and the code that
-runs the unit test for the DAG is in
-src/test/java/com/example/mydtapp/ApplicationTest.java. Try it out by
-running the following command:
-
-```
- $cd mydtapp; mvn package
-```
-This builds the App Package runs the unit test of the DAG. You should
-be getting test output similar to this:
-
-```
- -------------------------------------------------------
- TESTS
- -------------------------------------------------------
-
- Running com.example.mydtapp.ApplicationTest
- hello world: 0.8015370953286478
- hello world: 0.9785359225545481
- hello world: 0.6322611586644047
- hello world: 0.8460953663451775
- hello world: 0.5719372906929072
- hello world: 0.6361174312337172
- hello world: 0.14873007534816318
- hello world: 0.8866986277418261
- hello world: 0.6346526809866057
- hello world: 0.48587295703904465
- hello world: 0.6436832429676687
-
- ...
-
- Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.863
- sec
-
- Results :
-
- Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
-```
-
-The "mvn package" command creates the App Package file in target
-directory as target/mydtapp-1.0-SNAPSHOT.apa. You will be able to use
-that App Package file to launch this sample application in your actual
-Apex installation.
-
-## Using IDE
-
-Alternatively, you can do the above steps all within your IDE. For
-example, in NetBeans, select File -\> New Project. Then choose “Maven”
-and “Project from Archetype” in the dialog box, as shown.
-
-
-
-Then fill the Group ID, Artifact ID, Version and Repository entries as shown below.
-
-
-
-Group ID: com.datatorrent
-Artifact ID: apex-app-archetype
-Version: 3.0.0 (or any later version)
-
-Repository:
-[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases)
-
-Press Next and fill out the rest of the required information. For
-example:
-
-
-
-Click Finish, and now you have created your own DataTorrent App Package
-project, with a default unit test. You can run the unit test, make code
-changes or make dependency changes within your IDE. The procedure for
-other IDEs, like Eclipse or IntelliJ, is similar.
-
-# Writing Your Own App Package
-
-
-Please refer to the [Application Developer Guide](https://www.datatorrent.com/docs/guides/ApplicationDeveloperGuide.html) on the basics on how to write a DataTorrent application. In your AppPackage project, you can add custom operators (refer to [Operator Developer Guide](https://www.datatorrent.com/docs/guides/OperatorDeveloperGuide.html)), project dependencies, default and required configuration properties, pre-set configurations and other metadata.
-
-## Adding (and removing) project dependencies
-
-Under the project, you can add project dependencies in pom.xml, or do it
-through your IDE. Here’s the section that describes the dependencies in
-the default pom.xml:
-```
- <dependencies>
- <!-- add your dependencies here -->
- <dependency>
- <groupId>com.datatorrent</groupId>
- <artifactId>malhar-library</artifactId>
- <version>${datatorrent.version}</version>
- <!--
- If you know your application do not need the transitive dependencies that are pulled in by malhar-library,
- Uncomment the following to reduce the size of your app package.
- -->
- <!--
- <exclusions>
- <exclusion>
- <groupId>*</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- </exclusions>
- -->
- </dependency>
- <dependency>
- <groupId>com.datatorrent</groupId>
- <artifactId>dt-engine</artifactId>
- <version>${datatorrent.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>4.10</version>
- <scope>test</scope>
- </dependency>
- </dependencies>
-```
-
-By default, as shown above, the default dependencies include
-malhar-library in compile scope, dt-engine in provided scope, and junit
-in test scope. Do not remove these three dependencies since they are
-necessary for any Apex application. You can, however, exclude
-transitive dependencies from malhar-library to reduce the size of your
-App Package, provided that none of the operators in malhar-library that
-need the transitive dependencies will be used in your application.
-
-In the sample application, it is safe to remove the transitive
-dependencies from malhar-library, by uncommenting the "exclusions"
-section. It will reduce the size of the sample App Package from 8MB to
-700KB.
-
-Note that if we exclude \*, in some versions of Maven, you may get
-warnings similar to the following:
-
-```
-
- [WARNING] 'dependencies.dependency.exclusions.exclusion.groupId' for
- com.datatorrent:malhar-library:jar with value '*' does not match a
- valid id pattern.
-
- [WARNING]
- [WARNING] It is highly recommended to fix these problems because they
- threaten the stability of your build.
- [WARNING]
- [WARNING] For this reason, future Maven versions might no longer support
- building such malformed projects.
- [WARNING]
-
-```
-This is a bug in early versions of Maven 3. The dependency exclusion is
-still valid and it is safe to ignore these warnings.
-
-## Application Configuration
-
-A configuration file can be used to configure an application. Different
-kinds of configuration parameters can be specified. They are application
-attributes, operator attributes and properties, port attributes, stream
-properties and application specific properties. They are all specified
-as name value pairs, in XML format, like the following.
-
-```
-<?xml version="1.0"?>
-<configuration>
- <property>
- <name>some_name_1</name>
- <value>some_default_value</value>
- </property>
- <property>
- <name>some_name_2</name>
- <value>some_default_value</value>
- </property>
-</configuration>
-```
-
-## Application attributes
-
-Application attributes are used to specify the platform behavior for the
-application. They can be specified using the parameter
-```dt.attr.<attribute>```. The prefix “dt” is a constant, “attr” is a
-constant denoting an attribute is being specified and ```<attribute>```
-specifies the name of the attribute. Below is an example snippet setting
-the streaming windows size of the application to be 1000 milliseconds.
-
-```
- <property>
- <name>dt.attr.STREAMING_WINDOW_SIZE_MILLIS</name>
- <value>1000</value>
- </property>
-```
-
-The name tag specifies the attribute and value tag specifies the
-attribute value. The name of the attribute is a JAVA constant name
-identifying the attribute. The constants are defined in
-com.datatorrent.api.Context.DAGContext and the different attributes can
-be specified in the format described above.
-
-## Operator attributes
-
-Operator attributes are used to specify the platform behavior for the
-operator. They can be specified using the parameter
-```dt.operator.<operator-name>.attr.<attribute>```. The prefix “dt” is a
-constant, “operator” is a constant denoting that an operator is being
-specified, ```<operator-name>``` denotes the name of the operator, “attr” is
-the constant denoting that an attribute is being specified and
-```<attribute>``` is the name of the attribute. The operator name is the
-same name that is specified when the operator is added to the DAG using
-the addOperator method. An example illustrating the specification is
-shown below. It specifies the number of streaming windows for one
-application window of an operator named “input” to be 10
-
-```
-<property>
- <name>dt.operator.input.attr.APPLICATION_WINDOW_COUNT</name>
- <value>10</value>
-</property>
-```
-
-The name tag specifies the attribute and value tag specifies the
-attribute value. The name of the attribute is a JAVA constant name
-identifying the attribute. The constants are defined in
-com.datatorrent.api.Context.OperatorContext and the different attributes
-can be specified in the format described above.
-
-## Operator properties
-
-Operators can be configured using operator specific properties. The
-properties can be specified using the parameter
-```dt.operator.<operator-name>.prop.<property-name>```. The difference
-between this and the operator attribute specification described above is
-that the keyword “prop” is used to denote that it is a property and
-```<property-name>``` specifies the property name. An example illustrating
-this is specified below. It specifies the property “hostname” of the
-redis server for a “redis” output operator.
-
-```
- <property>
- <name>dt.operator.redis.prop.host</name>
- <value>127.0.0.1</value>
- </property>
-```
-
-The name tag specifies the property and the value specifies the property
-value. The property name is converted to a setter method which is called
-on the actual operator. The method name is composed by appending the
-word “set” and the property name with the first character of the name
-capitalized. In the above example the setter method would become
-setHost. The method is called using JAVA reflection and the property
-value is passed as an argument. In the above example the method setHost
-will be called on the “redis” operator with “127.0.0.1” as the argument.
-
-## Port attributes
-Port attributes are used to specify the platform behavior for input and
-output ports. They can be specified using the parameter ```dt.operator.<operator-name>.inputport.<port-name>.attr.<attribute>```
-for input port and ```dt.operator.<operator-name>.outputport.<port-name>.attr.<attribute>```
-for output port. The keyword “inputport” is used to denote an input port
-and “outputport” to denote an output port. The rest of the specification
-follows the conventions described in other specifications above. An
-example illustrating this is specified below. It specifies the queue
-capacity for an input port named “input” of an operator named “range” to
-be 4k.
-
-```
-<property>
- <name>dt.operator.range.inputport.input.attr.QUEUE_CAPACITY</name>
- <value>4000</value>
-</property>
-```
-
-The name tag specifies the attribute and value tag specifies the
-attribute value. The name of the attribute is a JAVA constant name
-identifying the attribute. The constants are defined in
-com.datatorrent.api.Context.PortContext and the different attributes can
-be specified in the format described above.
-
-The attributes for an output port can also be specified in a similar way
-as described above with a change that keyword “outputport” is used
-instead of “intputport”. A generic keyword “port” can be used to specify
-either an input or an output port. It is useful in the wildcard
-specification described below.
-
-## Stream properties
-
-Streams can be configured using stream properties. The properties can be
-specified using the parameter
-```dt.stream.<stream-name>.prop.<property-name>``` The constant “stream”
-specifies that it is a stream, ```<stream-name>``` specifies the name of the
-stream and ```<property-name>``` the name of the property. The name of the
-stream is the same name that is passed when the stream is added to the
-DAG using the addStream method. An example illustrating the
-specification is shown below. It sets the locality of the stream named
-“stream1” to container local indicating that the operators the stream is
-connecting be run in the same container.
-
-```
- <property>
- <name>dt.stream.stream1.prop.locality</name>
- <value>CONTAINER_LOCAL</value>
- </property>
-```
-
-The property name is converted into a set method on the stream in the
-same way as described in operator properties section above. In this case
-the method would be setLocality and it will be called in the stream
-“stream1” with the value as the argument.
-
-Along with the above system defined parameters, the applications can
-define their own specific parameters they can be specified in the
-configuration file. The only condition is that the names of these
-parameters don’t conflict with the system defined parameters or similar
-application parameters defined by other applications. To this end, it is
-recommended that the application parameters have the format
-```<full-application-class-name>.<param-name>.``` The
-full-application-class-name is the full JAVA class name of the
-application including the package path and param-name is the name of the
-parameter within the application. The application will still have to
-still read the parameter in using the configuration API of the
-configuration object that is passed in populateDAG.
-
-## Wildcards
-
-Wildcards and regular expressions can be used in place of names to
-specify a group for applications, operators, ports or streams. For
-example, to specify an attribute for all ports of an operator it can be
-done as follows
-```
-<property>
- <name>dt.operator.range.port.*.attr.QUEUE_CAPACITY</name>
- <value>4000</value>
-</property>
-```
-
-The wildcard “\*” was used instead of the name of the port. Wildcard can
-also be used for operator name, stream name or application name. Regular
-expressions can also be used for names to specify attributes or
-properties for a specific set.
-
-## Adding configuration properties
-
-It is common for applications to require configuration parameters to
-run. For example, the address and port of the database, the location of
-a file for ingestion, etc. You can specify them in
-src/main/resources/META-INF/properties.xml under the App Package
-project. The properties.xml may look like:
-
-```
-<?xml version="1.0"?>
-<configuration>
- <property>
- <name>some_name_1</name>
- </property>
- <property>
- <name>some_name_2</name>
- <value>some_default_value</value>
- </property>
-</configuration>
-```
-
-The name of an application-specific property takes the form of:
-
-```dt.operator.{opName}.prop.{propName} ```
-
-The first represents the property with name propName of operator opName.
- Or you can set the application name at run time by setting this
-property:
-
- dt.attr.APPLICATION_NAME
-
-There are also other properties that can be set. For details on
-properties, refer to the [Operation and Installation Guide](https://www.datatorrent.com/docs/guides/OperationandInstallationGuide.html).
-
-In this example, property some_name_1 is a required property which
-must be set at launch time, or it must be set by a pre-set configuration
-(see next section). Property some\_name\_2 is a property that is
-assigned with value some\_default\_value unless it is overridden at
-launch time.
-
-## Adding pre-set configurations
-
-
-At build time, you can add pre-set configurations to the App Package by
-adding configuration XML files under ```src/site/conf/<conf>.xml```in your
-project. You can then specify which configuration to use at launch
-time. The configuration XML is of the same format of the properties.xml
-file.
-
-## Application-specific properties file
-
-You can also specify properties.xml per application in the application
-package. Just create a file with the name properties-{appName}.xml and
-it will be picked up when you launch the application with the specified
-name within the application package. In short:
-
- properties.xml: Properties that are global to the Configuration
-Package
-
- properties-{appName}.xml: Properties that are specific when launching
-an application with the specified appName.
-
-## Properties source precedence
-
-If properties with the same key appear in multiple sources (e.g. from
-app package default configuration as META-INF/properties.xml, from app
-package configuration in the conf directory, from launch time defines,
-etc), the precedence of sources, from highest to lowest, is as follows:
-
-1. Launch time defines (using -D option in CLI, or the POST payload
- with the Gateway REST API’s launch call)
-2. Launch time specified configuration file in file system (using -conf
- option in CLI)
-3. Launch time specified package configuration (using -apconf option in
- CLI or the conf={confname} with Gateway REST API’s launch call)
-4. Configuration from \$HOME/.dt/dt-site.xml
-5. Application defaults within the package as
- META-INF/properties-{appname}.xml
-6. Package defaults as META-INF/properties.xml
-7. dt-site.xml in local DT installation
-8. dt-site.xml stored in HDFS
-
-## Other meta-data
-
-In a Apex App Package project, the pom.xml file contains a
-section that looks like:
-
-```
-<properties>
- <datatorrent.version>3.0.0</datatorrent.version>
- <datatorrent.apppackage.classpath\>lib*.jar</datatorrent.apppackage.classpath>
-</properties>
-```
-datatorrent.version is the DataTorrent RTS version that are to be used
-with this Application Package.
-
-datatorrent.apppackage.classpath is the classpath that is used when
-launching the application in the Application Package. The default is
-lib/\*.jar, where lib is where all the dependency jars are kept within
-the Application Package. One reason to change this field is when your
-Application Package needs the classpath in a specific order.
-
-## Logging configuration
-
-Just like other Java projects, you can change the logging configuration
-by having your log4j.properties under src/main/resources. For example,
-if you have the following in src/main/resources/log4j.properties:
-```
- log4j.rootLogger=WARN,CONSOLE
- log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
- log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
- log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [%t] %-5p
- %c{2} %M - %m%n
-```
-
-The root logger’s level is set to WARN and the output is set to the console (stdout).
-
-Note that by default from project created from the maven archetype,
-there is already a log4j.properties file under src/test/resources and
-that file is only used for the unit test.
-
-# Zip Structure of Application Package
-
-
-DataTorrent Application Package files are zip files. You can examine the content of any Application Package by using unzip -t on your Linux command line.
-
-There are four top level directories in an Application Package:
-
-1. "app" contains the jar files of the DAG code and any custom operators.
-2. "lib" contains all dependency jars
-3. "conf" contains all the pre-set configuration XML files.
-4. "META-INF" contains the MANIFEST.MF file and the properties.xml file.
-5. “resources” contains other files that are to be served by the Gateway on behalf of the app package.
-
-
-# Managing Application Packages Through DT Gateway
-
-The DT Gateway provides storing and retrieving Application Packages to
-and from your distributed file system, e.g. HDFS.
-
-## Storing an Application Package
-
-You can store your Application Packages through DT Gateway using this
-REST call:
-
-```
- POST /ws/v2/appPackages
-```
-
-The payload is the raw content of your Application Package. For
-example, you can issue this request using curl on your Linux command
-line like this, assuming your DT Gateway is accepting requests at
-localhost:9090:
-
-```
-$ curl -XPOST -T <app-package-file> http://localhost:9090/ws/v2/appPackages
-```
-
-## Getting Meta Information on Application Packages
-
-
-You can get the meta information on Application Packages stored through
-DT Gateway using this call. The information includes the logical plan
-of each application within the Application Package.
-
-```
- GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}
-```
-
-## Getting Available Operators In Application Package
-
-You can get the list of available operators in the Application Package
-using this call.
-
-```
-GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators?parent={parent}
-```
-
-The parent parameter is optional. If given, parent should be the fully
-qualified class name. It will only return operators that derive from
-that class or interface. For example, if parent is
-com.datatorrent.api.InputOperator, this call will only return input
-operators provided by the Application Package.
-
-## Getting Properties of Operators in Application Package
-
-You can get the list of properties of any operator in the Application
-Package using this call.
-
-```
-GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators/{className}
-```
-
-## Getting List of Pre-Set Configurations in Application Package
-
-You can get a list of pre-set configurations within the Application
-Package using this call.
-
-```
-GET /ws/v2/appPackages/{owner}/{pkgName}/{packageVersion}/configs
-```
-
-You can also get the content of a specific pre-set configuration within
-the Application Package.
-
-```
- GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
-```
-
-## Changing Pre-Set Configurations in Application Package
-
-You can create or replace pre-set configurations within the Application
-Package
-```
- PUT /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
-```
-The payload of this PUT call is the XML file that represents the pre-set configuration. The Content-Type of the payload is "application/xml" and you can delete a pre-set configuration within the Application Package.
-```
- DELETE /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
-```
-
-## Retrieving an Application Package
-
-You can download the Application Package file. This Application Package
-is not necessarily the same file as the one that was originally uploaded
-since the pre-set configurations may have been modified.
-
-```
- GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/download
-```
-
-## Launching an Application Package
-
-You can launch an application within an Application Package.
-```
-POST /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/applications/{appName}/launch?config={configName}
-```
-
-The config parameter is optional. If given, it must be one of the
-pre-set configuration within the given Application Package. The
-Content-Type of the payload of the POST request is "application/json"
-and should contain the properties to be launched with the application.
- It is of the form:
-
-```
- {"property-name":"property-value", ... }
-```
-
-Here is an example of launching an application through curl:
-
-```
- $ curl -XPOST -d'{"dt.operator.console.prop.stringFormat":"xyz %s"}'
- http://localhost:9090/ws/v2/appPackages/dtadmin/mydtapp/1.0-SNAPSHOT/app
- lications/MyFirstApplication/launch
-```
-
-Please refer to the [Gateway API reference](https://www.google.com/url?q=https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html&sa=D&usg=AFQjCNEWfN7-e7fd6MoWZjmJUE3GW7UwdQ) for the complete specification of the REST API.
-
-# Examining and Launching Application Packages Through Apex CLI
-
-If you are working with Application Packages in the local filesystem and
-do not want to deal with dtGateway, you can use the Apex
-Command Line Interface (dtcli). Please refer to the [Gateway API reference](https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html)
-to see samples for these commands.
-
-## Getting Application Package Meta Information
-
-You can get the meta information about the Application Package using
-this Apex CLI command.
-
-```
- dt> get-app-package-info <app-package-file>
-```
-
-## Getting Available Operators In Application Package
-
-You can get the list of available operators in the Application Package
-using this command.
-
-```
- dt> get-app-package-operators <app-package-file> <package-prefix>
- [parent-class]
-```
-
-## Getting Properties of Operators in Application Package
-
-You can get the list of properties of any operator in the Application
-Package using this command.
-
- dt> get-app-package-operator-properties <app-package-file> <operator-class>
-
-
-## Launching an Application Package
-
-You can launch an application within an Application Package.
-```
-dt> launch [-D property-name=property-value, ...] [-conf config-name]
- [-apconf config-file-within-app-package] <app-package-file>
- [matching-app-name]
- ```
-Note that -conf expects a configuration file in the file system, while -apconf expects a configuration file within the app package.
-
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/1cbcb0f4/AppConfPackage.md
----------------------------------------------------------------------
diff --git a/AppConfPackage.md b/AppConfPackage.md
deleted file mode 100644
index ef75b8d..0000000
--- a/AppConfPackage.md
+++ /dev/null
@@ -1,246 +0,0 @@
-#Application Configuration Packages
-
-# Introduction
-
-An Apex Application Configuration Package is a zip file that contains
-configuration files and additional files to be launched with an
-[Application Package](https://www.datatorrent.com/docs/guides/ApplicationPackages.html) using
-DTCLI or REST API. This guide assumes the reader’s familiarity of
-Application Package. Please read the Application Package document to
-get yourself familiar with the concept first if you have not done so.
-
-#Requirements
-
-You will need have the following installed:
-
-1. Apache Maven 3.0 or later (for assembling the Config Package)
-2. Apex 3.0.0 or later (for launching the App Package with the Config
- Package in your cluster)
-
-#Creating Your First Configuration Package
-
-You can create a Configuration Package using your Linux command line, or
-using your favorite IDE.
-
-## Using Command Line
-
-First, change to the directory where you put your projects, and create a
-DT configuration project using Maven by running the following command.
- Replace "com.example", "mydtconfig" and "1.0-SNAPSHOT" with the
-appropriate values:
-
-```
- $ mvn archetype:generate
- -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito
- ries/releases
- -DarchetypeGroupId=com.datatorrent
- -DarchetypeArtifactId=apex-conf-archetype -DarchetypeVersion=3.0.0
- -DgroupId=com.example -Dpackage=com.example.mydtconfig
- -DartifactId=mydtconfig -Dversion=1.0-SNAPSHOT
-
-```
-
-This creates a Maven project named "mydtconfig". Open it with your
-favorite IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). Try it out by
-running the following command:
-```
-$ mvn package
-```
-
-The "mvn package" command creates the Config Package file in target
-directory as target/mydtconfig.apc. You will be able to use that
-Configuration Package file to launch an application in your actual
-DataTorrent RTS installation.
-
-## Using IDE
-
-Alternatively, you can do the above steps all within your IDE. For
-example, in NetBeans, select File -\> New Project. Then choose “Maven”
-and “Project from Archetype” in the dialog box, as shown.
-
-
-
-Then fill the Group ID, Artifact ID, Version and Repository entries as
-shown below.
-
-
-
-Group ID: com.datatorrent
-Artifact ID: apex-conf-archetype
-Version: 3.0.0 (or any later version)
-
-Repository:
-[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases)
-
-[](https://www.datatorrent.com/maven/content/repositories/releases)
-
-Press Next and fill out the rest of the required information. For
-example:
-
-
-
-Click Finish, and now you have created your own Apex
-Configuration Package project. The procedure for other IDEs, like
-Eclipse or IntelliJ, is similar.
-
-#Assembling your own configuration package
-
-Inside the project created by the archetype, these are the files that
-you should know about when assembling your own configuration package:
-
- ./pom.xml
- ./src/main/resources/classpath
- ./src/main/resources/files
-
-./src/main/resources/META-INF/properties.xml
-./src/main/resources/META-INF/properties-{appname}.xml
-
-##pom.xml
-
-Example:
-
-```
- <groupId>com.example</groupId>
- <version>1.0.0</version>
- <artifactId>mydtconf</artifactId>
- <packaging>jar</packaging>
- <!-- change these to the appropriate values -->
- <name>My DataTorrent Application Configuration</name>
- <description>My DataTorrent Application Configuration Description</description>
- <properties>
- <datatorrent.apppackage.name>mydtapp</datatorrent.apppackage.name>
- <datatorrent.apppackage.minversion>1.0.0</datatorrent.apppackage.minversion>
- <datatorrent.apppackage.maxversion>1.9999.9999</datatorrent.apppackage.maxversion>
- <datatorrent.appconf.classpath>classpath/*</datatorrent.appconf.classpath>
- <datatorrent.appconf.files>files/*</datatorrent.appconf.files>
- </properties>
-
-```
-In pom.xml, you can change the values of ```<groupId>, <version>,
-<artifactId>, <name> ```and ```<description>``` to your desired values.
-
-You can also change the values of ```<datatorrent.apppackage.name>, <datatorrent.apppackage.minversion> <datatorrent.apppackage.maxversion>```
-to reflect what app packages should be used with this configuration
-package. Apex will use this information to check whether a
-configuration package is compatible with the application package when
-you issue a launch command.
-
-## ./src/main/resources/classpath
-
-Place any file in this directory that you’d like to be copied to the
-compute machines when launching an application and included in the
-classpath of the application. Example of such files are Java properties
-files and jar files.
-
-## ./src/main/resources/files
-
-Place any file in this directory that you’d like to be copied to the
-compute machines when launching an application but not included in the
-classpath of the application.
-
-## Properties XML file
-
-A properties xml file consists of a set of key-value pairs. The set of
-key-value pairs specifies the configuration options the application
-should be launched with.
-
-Example:
-```
-<configuration>
- <property>
- <name>some-property-name</name>
- <value>some-property-value</value>
- </property>
- ...
-</configuration>
-```
-Names of properties XML file:
-
- **properties.xml:** Properties that are global to the Configuration
-Package
-
- **properties-{appName}.xml:** Properties that are specific when launching
-an application with the specified appName within the Application
-Package.
-
-After you are done with the above, remember to do mvn package to
-generate a new configuration package, which will be located in the
-target directory in your project.
-
-## Zip structure of configuration package
-Apex Application Configuration Package files are zip files. You
-can examine the content of any Application Configuration Package by
-using unzip -t on your Linux command line. The structure of the zip
-file is as follow:
-
-```
-META-INF
- MANIFEST.MF
- properties.xml
- properties-{appname}.xml
-classpath
- {classpath files}
-files
- {files}
-```
-
-#Launching with CLI
-
--conf of the launch command in CLI supports specifying configuration
-package in the local filesystem. Example:
-
-dt\> launch DTApp-mydtapp-1.0.0.jar -conf DTConfig-mydtconfig-1.0.0.jar
-
-This command expects both the application package and the configuration
-package to be in the local file system.
-
-#Related REST API
-
-##POST /ws/v2/configPackages
-
-Payload: Raw content of configuration package zip
-Function: Creates or replace a configuration package zip file in HDFS
-
-Curl example:
-$ curl -XPOST -T DTConfig-{name}.jar http://{yourhost:port}/ws/v2/configPackages
-
-## GET /ws/v2/configPackages?appPackageName=...&appPackageVersion=...
-
-All query parameters are optional
-
-Function: Returns the configuration packages that the user is authorized
-to use and that are compatible with the specified appPackageName,
-appPackageVersion and appName.
-
-## GET /ws/v2/configPackages/``<user>``?appPackageName=...&appPackageVersion=...
-
-All query parameters are optional
-
-Function: Returns the configuration packages under the specified user
-and that are compatible with the specified appPackageName,
-appPackageVersion and appName.
-
-## GET /ws/v2/configPackages/```<user>```/```<name>```
-
-Function: Returns the information of the specified configuration package
-
-## GET /ws/v2/configPackages/```<user>```/```<name>```/download
-
-Function: Returns the raw config package file
-
-Curl example:
-$ curl http://{yourhost:port}/ws/v2/configPackages/{user}/{name}/download \> DTConfig-xyz.jar
-
-\$ unzip -t DTConfig-xyz.jar
-
-## POST /ws/v2/appPackages/```<user>```/```<app-pkg-name>```/```<app-pkg-version>```/applications/{app-name}/launch?configPackage=```<user>```/```<confpkgname>```
-
-Function: Launches the app package with the specified configuration
-package stored in HDFS.
-
-Curl example:
-
-$ curl -XPOST -d ’{}’ http://{yourhost:port}/ws/v2/appPackages/{user}/{app-pkg-name}/{app-pkg-version}/applications/{app-name}/launch?configPackage={user}/{confpkgname}
-
-© 2014-2015 DataTorrent Inc.
-
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/1cbcb0f4/apex_development_setup.md
----------------------------------------------------------------------
diff --git a/apex_development_setup.md b/apex_development_setup.md
new file mode 100644
index 0000000..fe98629
--- /dev/null
+++ b/apex_development_setup.md
@@ -0,0 +1,153 @@
+Apache Apex Development Environment Setup
+=========================================
+
+This document discusses the steps needed for setting up a development environment for creating applications that run on the Apache Apex or the DataTorrent RTS streaming platform.
+
+
+Microsoft Windows
+------------------------------
+
+There are a few tools that will be helpful when developing Apache Apex applications, some required and some optional:
+
+1. *git* -- A revision control system (version 1.7.1 or later). There are multiple git clients available for Windows ( http://git-scm.com/download/win for example), so download and install a client of your choice.
+
+2. *java JDK* (not JRE). Includes the Java Runtime Environment as well as the Java compiler and a variety of tools (version 1.7.0\_79 or later). Can be downloaded from the Oracle website.
+
+3. *maven* -- Apache Maven is a build system for Java projects (version 3.0.5 or later). It can be downloaded from https://maven.apache.org/download.cgi.
+
+4. *VirtualBox* -- Oracle VirtualBox is a virtual machine manager (version 4.3 or later) and can be downloaded from https://www.virtualbox.org/wiki/Downloads. It is needed to run the Data Torrent Sandbox.
+
+5. *DataTorrent Sandbox* -- The sandbox be downloaded from https://www.datatorrent.com/download. It is useful for testing simple applications since it contains Apache Hadoop and Data Torrent RTS 3.1.1 pre-installed with a time-limited Enterprise License. If you already installed the RTS Enterprise Edition (evaluation or production license) on a cluster, you can use that setup for deployment and testing instead of the sandbox.
+
+6. (Optional) If you prefer to use an IDE (Integrated Development Environment) such as *NetBeans*, *Eclipse* or *IntelliJ*, install that as well.
+
+
+After installing these tools, make sure that the directories containing the executable files are in your PATH environment; for example, for the JDK executables like java and javac, the directory might be something like `C:\\Program Files\\Java\\jdk1.7.0\_80\\bin`; for git it might be `C:\\Program Files\\Git\\bin`; and for maven it might be `C:\\Users\\user\\Software\\apache-maven-3.3.3\\bin`. Open a console window and enter the command:
+
+ echo %PATH%
+
+to see the contents of the PATH variable and verify that the above directories are present. If not, you can change the value of the variable at Control Panel -> Advanced System Settings -> Advanced tab -> Environment Variables button.
+
+
+Now run the following commands and ensure that the output is something similar to that shown in the table below:
+
+
+<table>
+<colgroup>
+<col width="30%" />
+<col width="70%" />
+</colgroup>
+<tbody>
+<tr class="odd">
+<td align="left"><p>Command</p></td>
+<td align="left"><p>Output</p></td>
+</tr>
+<tr class="even">
+<td align="left"><p>javac -version</p></td>
+<td align="left"><p>javac 1.7.0_80</p></td>
+</tr>
+<tr class="odd">
+<td align="left"><p>java -version</p></td>
+<td align="left"><p>java version "1.7.0_80"</p>
+<p>Java(TM) SE Runtime Environment (build 1.7.0_80-b15)</p>
+<p>Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)</p></td>
+</tr>
+<tr class="even">
+<td align="left"><p>git --version</p></td>
+<td align="left"><p>git version 2.6.1.windows.1</p></td>
+</tr>
+<tr class="odd">
+<td align="left"><p>mvn --version</p></td>
+<td align="left"><p>Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T06:57:37-05:00)</p>
+<p>Maven home: C:\Users\ram\Software\apache-maven-3.3.3\bin\..</p>
+<p>Java version: 1.7.0_80, vendor: Oracle Corporation</p>
+<p>Java home: C:\Program Files\Java\jdk1.7.0_80\jre</p>
+<p>Default locale: en_US, platform encoding: Cp1252</p>
+<p>OS name: "windows 8", version: "6.2", arch: "amd64", family: "windows"</p></td>
+</tr>
+</tbody>
+</table>
+
+
+
+To install the sandbox, first download it from <https://www.datatorrent.com/download/> and import the downloaded file into VirtualBox. Once the import completes, you can select it and click the Start button to start the sandbox.
+
+
+The sandbox is configured with 6GB RAM; if your development machine has 16GB or more, you can increase the sandbox RAM to 8GB or more using the VirtualBox console. This will yield better performance and support larger applications. Additionally, you can change the network adapter from " NAT" to "Bridged Adapter"; this will allow you to login to the sandbox from your host machine using an ssh tool like PuTTY and also to transfer files to and from the host using pscp on Windows. Of course all such configuration must be done when when the sandbox is not running.
+
+
+You can choose to develop either directly on the sandbox or on your development machine. The advantage of the former is that most of the tools (e.g. jdk, git, maven) are pre-installed and also the package files created by your project are directly available to the Data Torrent tools such as dtManage and dtcli. The disadvantage is that the sandbox is a memory-limited environment so running a memory-hungry tool like a Java IDE on it may starve other applications of memory.
+
+
+You can now use the maven archetype to create a basic Apache Apex project as follows: Put these lines in a Windows command file called, for example, newapp.cmd and run it:
+
+ @echo off
+ @rem Script for creating a new application
+ setlocal
+ mvn archetype:generate ^
+ -DarchetypeRepository=https://www.datatorrent.com/maven/content/repositories/releases ^
+ -DarchetypeGroupId=com.datatorrent ^
+ -DarchetypeArtifactId=apex-app-archetype ^
+ -DarchetypeVersion=3.1.1 ^
+ -DgroupId=com.example ^
+ -Dpackage=com.example.myapexapp ^
+ -DartifactId=myapexapp ^
+ -Dversion=1.0-SNAPSHOT
+ endlocal
+
+
+
+The caret (^) at the end of some lines indicates that a continuation line follows. When you run this file, the properties will be displayed and you will be prompted with " Y: :"; just press Enter to complete the project generation.
+
+
+This command file also exists in the Data Torrent examples repository which you can check out with:
+
+ git clone https://github.com/DataTorrent/examples
+
+You will find the script under `examples\\tutorials\\topnwords\\scripts\\newapp.cmd`.
+
+You can also, if you prefer, use an IDE to generate the project as described in Section 3 of [Apex Packages](apex_package.md) but use the archetype version 3.1.1 instead of 3.0.0.
+
+
+
+When the run completes successfully, you should see a new directory named myapexapp containing a maven project for building a basic Apache Apex application. It includes 3 source files: Application.java, RandomNumberGenerator.java and ApplicationTest.java. You can now build the application by stepping into the new directory and running the appropriate maven command:
+
+ cd myapexapp
+ mvn clean package -DskipTests
+
+The build should create the application package file myapexapp/target/myapexapp-1.0-SNAPSHOT.apa. This file can then be uploaded to the Data Torrent GUI tool on the sandbox (called dtManage) and launched from there. It generates a stream of random numbers and then prints them out, each prefixed by the string `"hello world: "`. If you built this package on the host, you can transfer it to the sandbox using the pscp tool bundled with PuTTY mentioned earlier.
+
+
+If you want to checkout the Apache Apex source repositories and build them, you can do so by running the script build-apex.cmd located in the same place in the examples repository described above. The source repositories contain more substantial demo applications and the associated source code. Alternatively, if you do not want to use the script, you can follow these simple manual steps:
+
+
+1. Check out the source code repositories:
+
+ git clone<https://github.com/apache/incubator-apex-core>
+ git clone<https://github.com/apache/incubator-apex-malhar>
+
+2. Switch to the appropriate release branch and build each repository:
+
+ pushd incubator-apex-core
+ git checkout release-3.1
+ mvn clean install -DskipTests
+ popd
+ pushd incubator-apex-malhar
+ git checkout release-3.1
+ mvn clean install -DskipTests
+ popd
+
+The *install* argument to the mvn command installs resources from each project to your local maven repository (typically .m2/repository under your home directory), and NOT to the system directories, so Administrator privileges are not required. The `-DskipTests` argument skips running unit tests since they take a long time. If this is a first-time installation, it might take several minutes to complete because maven will download a number of associated plugins.
+
+After the build completes, you should see the demo application package files in the target directory under each demo subdirectory in `incubator-apex-malhar/demos/`.
+
+Linux
+------------------
+
+Most of the instructions for Linux (and other Unix-like systems) are similar to those for Windows described above, so we will just note the differences.
+
+
+The pre-requisites (such as git, maven, etc.) are the same as for Windows as described above; please run the commands in the table and ensure that appropriate versions are present in your PATH environment variable (the command to display that variable is: `echo $PATH`).
+
+
+The maven archetype command is the same except that continuation lines use a backslash (\\) instead of caret (^); the script for it is available in the same location and is named newapp (without the `.cmd` extension). The script to checkout and build the Apache Apex repositories is named *build-apex*.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/1cbcb0f4/application_packages.md
----------------------------------------------------------------------
diff --git a/application_packages.md b/application_packages.md
new file mode 100644
index 0000000..1d6f6e8
--- /dev/null
+++ b/application_packages.md
@@ -0,0 +1,684 @@
+Apache Apex Application Packages
+================================
+
+An Apache Apex Application Package is a zip file that contains all the
+necessary files to launch an application in Apache Apex. It is the
+standard way for assembling and sharing an Apache Apex application.
+
+# Requirements
+
+You will need have the following installed:
+
+1. Apache Maven 3.0 or later (for assembling the App Package)
+2. Apache Apex 3.0.0 or later (for launching the App Package in your cluster)
+
+# Creating Your First Apex App Package
+
+You can create an Apex Application Package using your Linux command
+line, or using your favorite IDE.
+
+## Using Command Line
+
+First, change to the directory where you put your projects, and create
+an Apex application project using Maven by running the following
+command. Replace "com.example", "mydtapp" and "1.0-SNAPSHOT" with the
+appropriate values (make sure this is all on one line):
+
+```
+ $ mvn archetype:generate
+ -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito
+ ries/releases
+ -DarchetypeGroupId=com.datatorrent
+ -DarchetypeArtifactId=apex-app-archetype -DarchetypeVersion=3.2.0
+ -DgroupId=com.example -Dpackage=com.example.mydtapp -DartifactId=mydtapp
+ -Dversion=1.0-SNAPSHOT
+```
+
+This creates a Maven project named "mydtapp". Open it with your favorite
+IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). In the project, there is a
+sample DAG that generates a number of tuples with a random number and
+prints out "hello world" and the random number in the tuples. The code
+that builds the DAG is in
+src/main/java/com/example/mydtapp/Application.java, and the code that
+runs the unit test for the DAG is in
+src/test/java/com/example/mydtapp/ApplicationTest.java. Try it out by
+running the following command:
+
+```
+ $cd mydtapp; mvn package
+```
+This builds the App Package runs the unit test of the DAG. You should
+be getting test output similar to this:
+
+```
+ -------------------------------------------------------
+ TESTS
+ -------------------------------------------------------
+
+ Running com.example.mydtapp.ApplicationTest
+ hello world: 0.8015370953286478
+ hello world: 0.9785359225545481
+ hello world: 0.6322611586644047
+ hello world: 0.8460953663451775
+ hello world: 0.5719372906929072
+ hello world: 0.6361174312337172
+ hello world: 0.14873007534816318
+ hello world: 0.8866986277418261
+ hello world: 0.6346526809866057
+ hello world: 0.48587295703904465
+ hello world: 0.6436832429676687
+
+ ...
+
+ Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.863
+ sec
+
+ Results :
+
+ Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
+```
+
+The "mvn package" command creates the App Package file in target
+directory as target/mydtapp-1.0-SNAPSHOT.apa. You will be able to use
+that App Package file to launch this sample application in your actual
+Apex installation.
+
+## Using IDE
+
+Alternatively, you can do the above steps all within your IDE. For
+example, in NetBeans, select File -\> New Project. Then choose “Maven”
+and “Project from Archetype” in the dialog box, as shown.
+
+
+
+Then fill the Group ID, Artifact ID, Version and Repository entries as shown below.
+
+
+
+Group ID: com.datatorrent
+Artifact ID: apex-app-archetype
+Version: 3.0.0 (or any later version)
+
+Repository:
+[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases)
+
+Press Next and fill out the rest of the required information. For
+example:
+
+
+
+Click Finish, and now you have created your own DataTorrent App Package
+project, with a default unit test. You can run the unit test, make code
+changes or make dependency changes within your IDE. The procedure for
+other IDEs, like Eclipse or IntelliJ, is similar.
+
+# Writing Your Own App Package
+
+
+Please refer to the [Application Developer Guide](https://www.datatorrent.com/docs/guides/ApplicationDeveloperGuide.html) on the basics on how to write a DataTorrent application. In your AppPackage project, you can add custom operators (refer to [Operator Developer Guide](https://www.datatorrent.com/docs/guides/OperatorDeveloperGuide.html)), project dependencies, default and required configuration properties, pre-set configurations and other metadata.
+
+## Adding (and removing) project dependencies
+
+Under the project, you can add project dependencies in pom.xml, or do it
+through your IDE. Here’s the section that describes the dependencies in
+the default pom.xml:
+```
+ <dependencies>
+ <!-- add your dependencies here -->
+ <dependency>
+ <groupId>com.datatorrent</groupId>
+ <artifactId>malhar-library</artifactId>
+ <version>${datatorrent.version}</version>
+ <!--
+ If you know your application do not need the transitive dependencies that are pulled in by malhar-library,
+ Uncomment the following to reduce the size of your app package.
+ -->
+ <!--
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ -->
+ </dependency>
+ <dependency>
+ <groupId>com.datatorrent</groupId>
+ <artifactId>dt-engine</artifactId>
+ <version>${datatorrent.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.10</version>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+```
+
+By default, as shown above, the default dependencies include
+malhar-library in compile scope, dt-engine in provided scope, and junit
+in test scope. Do not remove these three dependencies since they are
+necessary for any Apex application. You can, however, exclude
+transitive dependencies from malhar-library to reduce the size of your
+App Package, provided that none of the operators in malhar-library that
+need the transitive dependencies will be used in your application.
+
+In the sample application, it is safe to remove the transitive
+dependencies from malhar-library, by uncommenting the "exclusions"
+section. It will reduce the size of the sample App Package from 8MB to
+700KB.
+
+Note that if we exclude \*, in some versions of Maven, you may get
+warnings similar to the following:
+
+```
+
+ [WARNING] 'dependencies.dependency.exclusions.exclusion.groupId' for
+ com.datatorrent:malhar-library:jar with value '*' does not match a
+ valid id pattern.
+
+ [WARNING]
+ [WARNING] It is highly recommended to fix these problems because they
+ threaten the stability of your build.
+ [WARNING]
+ [WARNING] For this reason, future Maven versions might no longer support
+ building such malformed projects.
+ [WARNING]
+
+```
+This is a bug in early versions of Maven 3. The dependency exclusion is
+still valid and it is safe to ignore these warnings.
+
+## Application Configuration
+
+A configuration file can be used to configure an application. Different
+kinds of configuration parameters can be specified. They are application
+attributes, operator attributes and properties, port attributes, stream
+properties and application specific properties. They are all specified
+as name value pairs, in XML format, like the following.
+
+```
+<?xml version="1.0"?>
+<configuration>
+ <property>
+ <name>some_name_1</name>
+ <value>some_default_value</value>
+ </property>
+ <property>
+ <name>some_name_2</name>
+ <value>some_default_value</value>
+ </property>
+</configuration>
+```
+
+## Application attributes
+
+Application attributes are used to specify the platform behavior for the
+application. They can be specified using the parameter
+```dt.attr.<attribute>```. The prefix “dt” is a constant, “attr” is a
+constant denoting an attribute is being specified and ```<attribute>```
+specifies the name of the attribute. Below is an example snippet setting
+the streaming windows size of the application to be 1000 milliseconds.
+
+```
+ <property>
+ <name>dt.attr.STREAMING_WINDOW_SIZE_MILLIS</name>
+ <value>1000</value>
+ </property>
+```
+
+The name tag specifies the attribute and value tag specifies the
+attribute value. The name of the attribute is a JAVA constant name
+identifying the attribute. The constants are defined in
+com.datatorrent.api.Context.DAGContext and the different attributes can
+be specified in the format described above.
+
+## Operator attributes
+
+Operator attributes are used to specify the platform behavior for the
+operator. They can be specified using the parameter
+```dt.operator.<operator-name>.attr.<attribute>```. The prefix “dt” is a
+constant, “operator” is a constant denoting that an operator is being
+specified, ```<operator-name>``` denotes the name of the operator, “attr” is
+the constant denoting that an attribute is being specified and
+```<attribute>``` is the name of the attribute. The operator name is the
+same name that is specified when the operator is added to the DAG using
+the addOperator method. An example illustrating the specification is
+shown below. It specifies the number of streaming windows for one
+application window of an operator named “input” to be 10
+
+```
+<property>
+ <name>dt.operator.input.attr.APPLICATION_WINDOW_COUNT</name>
+ <value>10</value>
+</property>
+```
+
+The name tag specifies the attribute and value tag specifies the
+attribute value. The name of the attribute is a JAVA constant name
+identifying the attribute. The constants are defined in
+com.datatorrent.api.Context.OperatorContext and the different attributes
+can be specified in the format described above.
+
+## Operator properties
+
+Operators can be configured using operator specific properties. The
+properties can be specified using the parameter
+```dt.operator.<operator-name>.prop.<property-name>```. The difference
+between this and the operator attribute specification described above is
+that the keyword “prop” is used to denote that it is a property and
+```<property-name>``` specifies the property name. An example illustrating
+this is specified below. It specifies the property “hostname” of the
+redis server for a “redis” output operator.
+
+```
+ <property>
+ <name>dt.operator.redis.prop.host</name>
+ <value>127.0.0.1</value>
+ </property>
+```
+
+The name tag specifies the property and the value specifies the property
+value. The property name is converted to a setter method which is called
+on the actual operator. The method name is composed by appending the
+word “set” and the property name with the first character of the name
+capitalized. In the above example the setter method would become
+setHost. The method is called using JAVA reflection and the property
+value is passed as an argument. In the above example the method setHost
+will be called on the “redis” operator with “127.0.0.1” as the argument.
+
+## Port attributes
+Port attributes are used to specify the platform behavior for input and
+output ports. They can be specified using the parameter ```dt.operator.<operator-name>.inputport.<port-name>.attr.<attribute>```
+for input port and ```dt.operator.<operator-name>.outputport.<port-name>.attr.<attribute>```
+for output port. The keyword “inputport” is used to denote an input port
+and “outputport” to denote an output port. The rest of the specification
+follows the conventions described in other specifications above. An
+example illustrating this is specified below. It specifies the queue
+capacity for an input port named “input” of an operator named “range” to
+be 4k.
+
+```
+<property>
+ <name>dt.operator.range.inputport.input.attr.QUEUE_CAPACITY</name>
+ <value>4000</value>
+</property>
+```
+
+The name tag specifies the attribute and value tag specifies the
+attribute value. The name of the attribute is a JAVA constant name
+identifying the attribute. The constants are defined in
+com.datatorrent.api.Context.PortContext and the different attributes can
+be specified in the format described above.
+
+The attributes for an output port can also be specified in a similar way
+as described above with a change that keyword “outputport” is used
+instead of “intputport”. A generic keyword “port” can be used to specify
+either an input or an output port. It is useful in the wildcard
+specification described below.
+
+## Stream properties
+
+Streams can be configured using stream properties. The properties can be
+specified using the parameter
+```dt.stream.<stream-name>.prop.<property-name>``` The constant “stream”
+specifies that it is a stream, ```<stream-name>``` specifies the name of the
+stream and ```<property-name>``` the name of the property. The name of the
+stream is the same name that is passed when the stream is added to the
+DAG using the addStream method. An example illustrating the
+specification is shown below. It sets the locality of the stream named
+“stream1” to container local indicating that the operators the stream is
+connecting be run in the same container.
+
+```
+ <property>
+ <name>dt.stream.stream1.prop.locality</name>
+ <value>CONTAINER_LOCAL</value>
+ </property>
+```
+
+The property name is converted into a set method on the stream in the
+same way as described in operator properties section above. In this case
+the method would be setLocality and it will be called in the stream
+“stream1” with the value as the argument.
+
+Along with the above system defined parameters, the applications can
+define their own specific parameters they can be specified in the
+configuration file. The only condition is that the names of these
+parameters don’t conflict with the system defined parameters or similar
+application parameters defined by other applications. To this end, it is
+recommended that the application parameters have the format
+```<full-application-class-name>.<param-name>.``` The
+full-application-class-name is the full JAVA class name of the
+application including the package path and param-name is the name of the
+parameter within the application. The application will still have to
+still read the parameter in using the configuration API of the
+configuration object that is passed in populateDAG.
+
+## Wildcards
+
+Wildcards and regular expressions can be used in place of names to
+specify a group for applications, operators, ports or streams. For
+example, to specify an attribute for all ports of an operator it can be
+done as follows
+```
+<property>
+ <name>dt.operator.range.port.*.attr.QUEUE_CAPACITY</name>
+ <value>4000</value>
+</property>
+```
+
+The wildcard “\*” was used instead of the name of the port. Wildcard can
+also be used for operator name, stream name or application name. Regular
+expressions can also be used for names to specify attributes or
+properties for a specific set.
+
+## Adding configuration properties
+
+It is common for applications to require configuration parameters to
+run. For example, the address and port of the database, the location of
+a file for ingestion, etc. You can specify them in
+src/main/resources/META-INF/properties.xml under the App Package
+project. The properties.xml may look like:
+
+```
+<?xml version="1.0"?>
+<configuration>
+ <property>
+ <name>some_name_1</name>
+ </property>
+ <property>
+ <name>some_name_2</name>
+ <value>some_default_value</value>
+ </property>
+</configuration>
+```
+
+The name of an application-specific property takes the form of:
+
+```dt.operator.{opName}.prop.{propName} ```
+
+The first represents the property with name propName of operator opName.
+ Or you can set the application name at run time by setting this
+property:
+
+ dt.attr.APPLICATION_NAME
+
+There are also other properties that can be set. For details on
+properties, refer to the [Operation and Installation Guide](https://www.datatorrent.com/docs/guides/OperationandInstallationGuide.html).
+
+In this example, property some_name_1 is a required property which
+must be set at launch time, or it must be set by a pre-set configuration
+(see next section). Property some\_name\_2 is a property that is
+assigned with value some\_default\_value unless it is overridden at
+launch time.
+
+## Adding pre-set configurations
+
+
+At build time, you can add pre-set configurations to the App Package by
+adding configuration XML files under ```src/site/conf/<conf>.xml```in your
+project. You can then specify which configuration to use at launch
+time. The configuration XML is of the same format of the properties.xml
+file.
+
+## Application-specific properties file
+
+You can also specify properties.xml per application in the application
+package. Just create a file with the name properties-{appName}.xml and
+it will be picked up when you launch the application with the specified
+name within the application package. In short:
+
+ properties.xml: Properties that are global to the Configuration
+Package
+
+ properties-{appName}.xml: Properties that are specific when launching
+an application with the specified appName.
+
+## Properties source precedence
+
+If properties with the same key appear in multiple sources (e.g. from
+app package default configuration as META-INF/properties.xml, from app
+package configuration in the conf directory, from launch time defines,
+etc), the precedence of sources, from highest to lowest, is as follows:
+
+1. Launch time defines (using -D option in CLI, or the POST payload
+ with the Gateway REST API’s launch call)
+2. Launch time specified configuration file in file system (using -conf
+ option in CLI)
+3. Launch time specified package configuration (using -apconf option in
+ CLI or the conf={confname} with Gateway REST API’s launch call)
+4. Configuration from \$HOME/.dt/dt-site.xml
+5. Application defaults within the package as
+ META-INF/properties-{appname}.xml
+6. Package defaults as META-INF/properties.xml
+7. dt-site.xml in local DT installation
+8. dt-site.xml stored in HDFS
+
+## Other meta-data
+
+In a Apex App Package project, the pom.xml file contains a
+section that looks like:
+
+```
+<properties>
+ <datatorrent.version>3.0.0</datatorrent.version>
+ <datatorrent.apppackage.classpath\>lib*.jar</datatorrent.apppackage.classpath>
+</properties>
+```
+datatorrent.version is the DataTorrent RTS version that are to be used
+with this Application Package.
+
+datatorrent.apppackage.classpath is the classpath that is used when
+launching the application in the Application Package. The default is
+lib/\*.jar, where lib is where all the dependency jars are kept within
+the Application Package. One reason to change this field is when your
+Application Package needs the classpath in a specific order.
+
+## Logging configuration
+
+Just like other Java projects, you can change the logging configuration
+by having your log4j.properties under src/main/resources. For example,
+if you have the following in src/main/resources/log4j.properties:
+```
+ log4j.rootLogger=WARN,CONSOLE
+ log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
+ log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
+ log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [%t] %-5p
+ %c{2} %M - %m%n
+```
+
+The root logger’s level is set to WARN and the output is set to the console (stdout).
+
+Note that by default from project created from the maven archetype,
+there is already a log4j.properties file under src/test/resources and
+that file is only used for the unit test.
+
+# Zip Structure of Application Package
+
+
+DataTorrent Application Package files are zip files. You can examine the content of any Application Package by using unzip -t on your Linux command line.
+
+There are four top level directories in an Application Package:
+
+1. "app" contains the jar files of the DAG code and any custom operators.
+2. "lib" contains all dependency jars
+3. "conf" contains all the pre-set configuration XML files.
+4. "META-INF" contains the MANIFEST.MF file and the properties.xml file.
+5. “resources” contains other files that are to be served by the Gateway on behalf of the app package.
+
+
+# Managing Application Packages Through DT Gateway
+
+The DT Gateway provides storing and retrieving Application Packages to
+and from your distributed file system, e.g. HDFS.
+
+## Storing an Application Package
+
+You can store your Application Packages through DT Gateway using this
+REST call:
+
+```
+ POST /ws/v2/appPackages
+```
+
+The payload is the raw content of your Application Package. For
+example, you can issue this request using curl on your Linux command
+line like this, assuming your DT Gateway is accepting requests at
+localhost:9090:
+
+```
+$ curl -XPOST -T <app-package-file> http://localhost:9090/ws/v2/appPackages
+```
+
+## Getting Meta Information on Application Packages
+
+
+You can get the meta information on Application Packages stored through
+DT Gateway using this call. The information includes the logical plan
+of each application within the Application Package.
+
+```
+ GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}
+```
+
+## Getting Available Operators In Application Package
+
+You can get the list of available operators in the Application Package
+using this call.
+
+```
+GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators?parent={parent}
+```
+
+The parent parameter is optional. If given, parent should be the fully
+qualified class name. It will only return operators that derive from
+that class or interface. For example, if parent is
+com.datatorrent.api.InputOperator, this call will only return input
+operators provided by the Application Package.
+
+## Getting Properties of Operators in Application Package
+
+You can get the list of properties of any operator in the Application
+Package using this call.
+
+```
+GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators/{className}
+```
+
+## Getting List of Pre-Set Configurations in Application Package
+
+You can get a list of pre-set configurations within the Application
+Package using this call.
+
+```
+GET /ws/v2/appPackages/{owner}/{pkgName}/{packageVersion}/configs
+```
+
+You can also get the content of a specific pre-set configuration within
+the Application Package.
+
+```
+ GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
+```
+
+## Changing Pre-Set Configurations in Application Package
+
+You can create or replace pre-set configurations within the Application
+Package
+```
+ PUT /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
+```
+The payload of this PUT call is the XML file that represents the pre-set configuration. The Content-Type of the payload is "application/xml" and you can delete a pre-set configuration within the Application Package.
+```
+ DELETE /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName}
+```
+
+## Retrieving an Application Package
+
+You can download the Application Package file. This Application Package
+is not necessarily the same file as the one that was originally uploaded
+since the pre-set configurations may have been modified.
+
+```
+ GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/download
+```
+
+## Launching an Application Package
+
+You can launch an application within an Application Package.
+```
+POST /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/applications/{appName}/launch?config={configName}
+```
+
+The config parameter is optional. If given, it must be one of the
+pre-set configuration within the given Application Package. The
+Content-Type of the payload of the POST request is "application/json"
+and should contain the properties to be launched with the application.
+ It is of the form:
+
+```
+ {"property-name":"property-value", ... }
+```
+
+Here is an example of launching an application through curl:
+
+```
+ $ curl -XPOST -d'{"dt.operator.console.prop.stringFormat":"xyz %s"}'
+ http://localhost:9090/ws/v2/appPackages/dtadmin/mydtapp/1.0-SNAPSHOT/app
+ lications/MyFirstApplication/launch
+```
+
+Please refer to the [Gateway API reference](https://www.google.com/url?q=https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html&sa=D&usg=AFQjCNEWfN7-e7fd6MoWZjmJUE3GW7UwdQ) for the complete specification of the REST API.
+
+# Examining and Launching Application Packages Through Apex CLI
+
+If you are working with Application Packages in the local filesystem and
+do not want to deal with dtGateway, you can use the Apex
+Command Line Interface (dtcli). Please refer to the [Gateway API reference](https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html)
+to see samples for these commands.
+
+## Getting Application Package Meta Information
+
+You can get the meta information about the Application Package using
+this Apex CLI command.
+
+```
+ dt> get-app-package-info <app-package-file>
+```
+
+## Getting Available Operators In Application Package
+
+You can get the list of available operators in the Application Package
+using this command.
+
+```
+ dt> get-app-package-operators <app-package-file> <package-prefix>
+ [parent-class]
+```
+
+## Getting Properties of Operators in Application Package
+
+You can get the list of properties of any operator in the Application
+Package using this command.
+
+ dt> get-app-package-operator-properties <app-package-file> <operator-class>
+
+
+## Launching an Application Package
+
+You can launch an application within an Application Package.
+```
+dt> launch [-D property-name=property-value, ...] [-conf config-name]
+ [-apconf config-file-within-app-package] <app-package-file>
+ [matching-app-name]
+ ```
+Note that -conf expects a configuration file in the file system, while -apconf expects a configuration file within the app package.
+
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/1cbcb0f4/configuration_packages.md
----------------------------------------------------------------------
diff --git a/configuration_packages.md b/configuration_packages.md
new file mode 100644
index 0000000..27d3978
--- /dev/null
+++ b/configuration_packages.md
@@ -0,0 +1,237 @@
+Apache Apex Configuration Packages
+==================================
+
+An Apache Apex Application Configuration Package is a zip file that contains
+configuration files and additional files to be launched with an
+[Application Package](https://www.datatorrent.com/docs/guides/ApplicationPackages.html) using
+DTCLI or REST API. This guide assumes the reader’s familiarity of
+Application Package. Please read the Application Package document to
+get yourself familiar with the concept first if you have not done so.
+
+#Requirements
+
+You will need have the following installed:
+
+1. Apache Maven 3.0 or later (for assembling the Config Package)
+2. Apex 3.0.0 or later (for launching the App Package with the Config
+ Package in your cluster)
+
+#Creating Your First Configuration Package
+
+You can create a Configuration Package using your Linux command line, or
+using your favorite IDE.
+
+## Using Command Line
+
+First, change to the directory where you put your projects, and create a
+DT configuration project using Maven by running the following command.
+ Replace "com.example", "mydtconfig" and "1.0-SNAPSHOT" with the
+appropriate values:
+
+```
+ $ mvn archetype:generate
+ -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito
+ ries/releases
+ -DarchetypeGroupId=com.datatorrent
+ -DarchetypeArtifactId=apex-conf-archetype -DarchetypeVersion=3.0.0
+ -DgroupId=com.example -Dpackage=com.example.mydtconfig
+ -DartifactId=mydtconfig -Dversion=1.0-SNAPSHOT
+
+```
+
+This creates a Maven project named "mydtconfig". Open it with your
+favorite IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). Try it out by
+running the following command:
+```
+$ mvn package
+```
+
+The "mvn package" command creates the Config Package file in target
+directory as target/mydtconfig.apc. You will be able to use that
+Configuration Package file to launch an application in your actual
+DataTorrent RTS installation.
+
+## Using IDE
+
+Alternatively, you can do the above steps all within your IDE. For
+example, in NetBeans, select File -\> New Project. Then choose “Maven”
+and “Project from Archetype” in the dialog box, as shown.
+
+
+
+Then fill the Group ID, Artifact ID, Version and Repository entries as
+shown below.
+
+
+
+Group ID: com.datatorrent
+Artifact ID: apex-conf-archetype
+Version: 3.0.0 (or any later version)
+
+Repository:
+[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases)
+
+[](https://www.datatorrent.com/maven/content/repositories/releases)
+
+Press Next and fill out the rest of the required information. For
+example:
+
+
+
+Click Finish, and now you have created your own Apex
+Configuration Package project. The procedure for other IDEs, like
+Eclipse or IntelliJ, is similar.
+
+#Assembling your own configuration package
+
+Inside the project created by the archetype, these are the files that
+you should know about when assembling your own configuration package:
+
+ ./pom.xml
+ ./src/main/resources/classpath
+ ./src/main/resources/files
+
+./src/main/resources/META-INF/properties.xml
+./src/main/resources/META-INF/properties-{appname}.xml
+
+##pom.xml
+
+Example:
+
+```
+ <groupId>com.example</groupId>
+ <version>1.0.0</version>
+ <artifactId>mydtconf</artifactId>
+ <packaging>jar</packaging>
+ <!-- change these to the appropriate values -->
+ <name>My DataTorrent Application Configuration</name>
+ <description>My DataTorrent Application Configuration Description</description>
+ <properties>
+ <datatorrent.apppackage.name>mydtapp</datatorrent.apppackage.name>
+ <datatorrent.apppackage.minversion>1.0.0</datatorrent.apppackage.minversion>
+ <datatorrent.apppackage.maxversion>1.9999.9999</datatorrent.apppackage.maxversion>
+ <datatorrent.appconf.classpath>classpath/*</datatorrent.appconf.classpath>
+ <datatorrent.appconf.files>files/*</datatorrent.appconf.files>
+ </properties>
+
+```
+In pom.xml, you can change the values of ```<groupId>, <version>, <artifactId>, <name> ```and ```<description>``` to your desired values.
+
+You can also change the values of ```<datatorrent.apppackage.name>, <datatorrent.apppackage.minversion> <datatorrent.apppackage.maxversion>```
+to reflect what app packages should be used with this configuration package. Apex will use this information to check whether a
+configuration package is compatible with the application package when you issue a launch command.
+
+## ./src/main/resources/classpath
+
+Place any file in this directory that you’d like to be copied to the
+compute machines when launching an application and included in the
+classpath of the application. Example of such files are Java properties
+files and jar files.
+
+## ./src/main/resources/files
+
+Place any file in this directory that you’d like to be copied to the
+compute machines when launching an application but not included in the
+classpath of the application.
+
+## Properties XML file
+
+A properties xml file consists of a set of key-value pairs. The set of
+key-value pairs specifies the configuration options the application
+should be launched with.
+
+Example:
+```
+<configuration>
+ <property>
+ <name>some-property-name</name>
+ <value>some-property-value</value>
+ </property>
+ ...
+</configuration>
+```
+Names of properties XML file:
+
+* **properties.xml:** Properties that are global to the Configuration
+Package
+* **properties-{appName}.xml:** Properties that are specific when launching
+an application with the specified appName within the Application
+Package.
+
+After you are done with the above, remember to do mvn package to
+generate a new configuration package, which will be located in the
+target directory in your project.
+
+## Zip structure of configuration package
+Apex Application Configuration Package files are zip files. You
+can examine the content of any Application Configuration Package by
+using unzip -t on your Linux command line. The structure of the zip
+file is as follow:
+
+```
+META-INF
+ MANIFEST.MF
+ properties.xml
+ properties-{appname}.xml
+classpath
+ {classpath files}
+files
+ {files}
+```
+
+#Launching with CLI
+
+`-conf` option of the launch command in CLI supports specifying configuration package in the local filesystem. Example:
+
+ dt\> launch DTApp-mydtapp-1.0.0.jar -conf DTConfig-mydtconfig-1.0.0.jar
+
+This command expects both the application package and the configuration package to be in the local file system.
+
+#Related REST API
+
+##POST /ws/v2/configPackages
+
+Payload: Raw content of configuration package zip
+
+Function: Creates or replace a configuration package zip file in HDFS
+
+Curl example:
+
+ $ curl -XPOST -T DTConfig-{name}.jar http://{yourhost:port}/ws/v2/configPackages
+
+## GET /ws/v2/configPackages?appPackageName=...&appPackageVersion=...
+
+All query parameters are optional
+
+Function: Returns the configuration packages that the user is authorized to use and that are compatible with the specified appPackageName, appPackageVersion and appName.
+
+## GET /ws/v2/configPackages/``<user>``?appPackageName=...&appPackageVersion=...
+
+All query parameters are optional
+
+Function: Returns the configuration packages under the specified user and that are compatible with the specified appPackageName, appPackageVersion and appName.
+
+## GET /ws/v2/configPackages/```<user>```/```<name>```
+
+Function: Returns the information of the specified configuration package
+
+## GET /ws/v2/configPackages/```<user>```/```<name>```/download
+
+Function: Returns the raw config package file
+
+Curl example:
+
+```sh
+$ curl http://{yourhost:port}/ws/v2/configPackages/{user}/{name}/download \> DTConfig-xyz.jar
+$ unzip -t DTConfig-xyz.jar
+```
+
+## POST /ws/v2/appPackages/```<user>```/```<app-pkg-name>```/```<app-pkg-version>```/applications/{app-name}/launch?configPackage=```<user>```/```<confpkgname>```
+
+Function: Launches the app package with the specified configuration package stored in HDFS.
+
+Curl example:
+
+```sh
+$ curl -XPOST -d ’{}’ http://{yourhost:port}/ws/v2/appPackages/{user}/{app-pkg-name}/{app-pkg-version}/applications/{app-name}/launch?configPackage={user}/{confpkgname}
+```
\ No newline at end of file