You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@apex.apache.org by pr...@apache.org on 2016/03/29 23:50:31 UTC
[1/2] incubator-apex-core git commit: APEXCORE-409 #resolve added
documentation for app specification using json
Repository: incubator-apex-core
Updated Branches:
refs/heads/master a784bf622 -> 9e1721110
APEXCORE-409 #resolve added documentation for app specification using json
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/48e179dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/48e179dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/48e179dd
Branch: refs/heads/master
Commit: 48e179dd7ed9c814c0ed36cac0ee2e40bba5a092
Parents: 48709d9
Author: David Yan <da...@datatorrent.com>
Authored: Mon Mar 28 16:05:22 2016 -0700
Committer: David Yan <da...@datatorrent.com>
Committed: Tue Mar 29 14:31:35 2016 -0700
----------------------------------------------------------------------
docs/application_development.md | 68 ++++++++++++++++++++++++++++--------
docs/application_packages.md | 5 +--
2 files changed, 56 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/48e179dd/docs/application_development.md
----------------------------------------------------------------------
diff --git a/docs/application_development.md b/docs/application_development.md
index eb56e8e..a720378 100644
--- a/docs/application_development.md
+++ b/docs/application_development.md
@@ -240,8 +240,7 @@ public class StockTickInput implements InputOperator
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
- }
- else {
+ } else {
InputStream istream = method.getResponseBodyAsStream();
// Process response
InputStreamReader isr = new InputStreamReader(istream);
@@ -272,11 +271,9 @@ public class StockTickInput implements InputOperator
}
}
Thread.sleep(readIntervalMillis);
- }
- catch (InterruptedException ex) {
+ } catch (InterruptedException ex) {
logger.debug(ex.toString());
- }
- catch (IOException ex) {
+ } catch (IOException ex) {
logger.debug(ex.toString());
}
}
@@ -293,7 +290,7 @@ public class StockTickInput implements InputOperator
public void setOutputEvenIfZeroVolume(boolean outputEvenIfZeroVolume)
{
- this.outputEvenIfZeroVolume = outputEvenIfZeroVolume;
+ this.outputEvenIfZeroVolume = outputEvenIfZeroVolume;
}
}
@@ -658,6 +655,7 @@ scalability an application running in this single process mode is more
likely to encounter throughput bottlenecks. A distributed cluster is
recommended for benchmarking and production testing.
+
Hadoop Cluster
---------------------------
@@ -1085,7 +1083,7 @@ The platform provides a commandline tool called dtcli for managing applications
killing, viewing, etc.). This tool was already discussed above briefly
in the section entitled Running the Test Application. It will introspect
the jar file specified with the launch command for applications (classes
-that implement ApplicationFactory) or property files that define
+that implement ApplicationFactory) or properties files that define
applications. It will also deploy the dependency jar files from the
application package to the cluster.
@@ -1196,12 +1194,54 @@ public class Application implements StreamingApplication
}
```
+JSON File DAG Specification
+--------------------------
+In addition to Java, you can also specify the DAG using JSON, provided the operators in the DAG are present in the dependency jars. Create src/main/resources/app directory under your app package project, and put your JSON files there. This is the specification of a JSON file that specifies an application.
+Create a json file under src/main/resources/app, For example `myApplication.json`
+```
+{
+ "description": "{application description}",
+ "operators": [
+ {
+ "name": "{operator name}",
+ "class": "{fully qualified class name of the operator}",
+ "properties": {
+ "{property key}": "{property value}",
+ ...
+ }
+ }, ...
+ ],
+ "streams": [
+ {
+ "name": "{stream name}",
+ "source": {
+ "operatorName": "{source operator name}",
+ "portName": "{source operator output port name}"
+ }
+ "sinks": [
+ {
+ "operatorName": "{sink operator name}",
+ "portName": "{sink operator input port name}"
+ }, ...
+ ]
+ }, ...
+ ]
+}
+
+```
+- The name of the JSON file is taken as the name of the application.
+- The `description` field is the description of the application and is optional.
+- The `operators` field is the list of operators the application has. You can specifiy the name, the Java class, and the properties of each operator here.
+- The `streams` field is the list of streams that connects the operators together to form the DAG. Each stream consists of the stream name, the operator and port that it connects from, and the list of operators and ports that it connects to. Note that you can connect from *one* output port of an operator to *multiple* different input ports of different operators.
+
+In Apex Malhar, there is an [example](https://github.com/apache/incubator-apex-malhar/blob/master/demos/pi/src/main/resources/app/PiJsonDemo.json) in the Pi Demo doing just that.
-### Property File API
-The platform also supports specification of a DAG via a property
+### Properties File DAG Specification
+
+The platform also supports specification of a DAG via a properties
file. The aim here to make it easy for tools to create and run an
application. This method of specification does not have the Java
compiler support of compile time check, but since these applications
@@ -1210,10 +1250,8 @@ The syntax is derived from Hadoop properties and should be easy for
folks who are used to creating software that integrated with
Hadoop.
-
-
-Create an application (DAG): myApplication.properties
-
+Under the src/main/resources/app directory (create if it doesn't exist), create a properties file.
+For example `myApplication.properties`
```
# input operator that reads from a file
@@ -1231,7 +1269,7 @@ dt.stream.inputStream.sinks=outputOp.inputPort
Above snippet is intended to convey the basic idea of specifying
-the DAG without using Java. Operators would come from a predefined
+the DAG using properties file. Operators would come from a predefined
library and referenced in the specification by class name and port names
(obtained from the library providers documentation or runtime
introspection by tools). For those interested in details, see later
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/48e179dd/docs/application_packages.md
----------------------------------------------------------------------
diff --git a/docs/application_packages.md b/docs/application_packages.md
index 1d1d937..06f5696 100644
--- a/docs/application_packages.md
+++ b/docs/application_packages.md
@@ -89,7 +89,7 @@ Version: 3.2.0-incubating (or any later version)
## Writing Your Own App Package
-Please refer to the [Creating Apps](http://docs.datatorrent.com/create/) on the basics on how to write an Apache Apex application. In your AppPackage project, you can add custom operators (refer to [Operator Development Guide](operator_development.md), project dependencies, default and required configuration properties, pre-set configurations and other metadata.
+Please refer to the [Application Developer Guide][application_development.md] on the basics on how to write an Apache Apex application. In your AppPackage project, you can add custom operators (refer to [Operator Development Guide](operator_development.md), project dependencies, default and required configuration properties, pre-set configurations and other metadata. Note that you can also specify the DAG using Java, JSON or properties files.
### Adding (and removing) project dependencies
@@ -467,6 +467,7 @@ Note that by default from project created from the maven archetype,
there is already a log4j.properties file under src/test/resources and
that file is only used for the unit test.
+
## Zip Structure of Application Package
@@ -474,7 +475,7 @@ Apache Apex Application Package files are zip files. You can examine the conten
There are four top level directories in an Application Package:
-1. "app" contains the jar files of the DAG code and any custom operators.
+1. "app" contains the jar files of the DAG code and any custom operators, and any JSON or properties files that specify a DAG.
2. "lib" contains all dependency jars
3. "conf" contains all the pre-set configuration XML files.
4. "META-INF" contains the MANIFEST.MF file and the properties.xml file.
[2/2] incubator-apex-core git commit: Merge branch 'APEXCORE-409' of
github.com:davidyan74/incubator-apex-core
Posted by pr...@apache.org.
Merge branch 'APEXCORE-409' of github.com:davidyan74/incubator-apex-core
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/9e172111
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/9e172111
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/9e172111
Branch: refs/heads/master
Commit: 9e172111014611baf89069fd462c097edcc24a0e
Parents: a784bf6 48e179d
Author: Pramod Immaneni <pr...@datatorrent.com>
Authored: Tue Mar 29 14:49:09 2016 -0700
Committer: Pramod Immaneni <pr...@datatorrent.com>
Committed: Tue Mar 29 14:49:09 2016 -0700
----------------------------------------------------------------------
docs/application_development.md | 68 ++++++++++++++++++++++++++++--------
docs/application_packages.md | 5 +--
2 files changed, 56 insertions(+), 17 deletions(-)
----------------------------------------------------------------------