You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@apex.apache.org by da...@apache.org on 2016/04/15 22:07:37 UTC
[4/5] incubator-apex-core git commit: APEXCORE-340 Naming change of
apex script. Changed following (dt->apex): 1. Script name changed to apex. 2.
Class name changed to ApexCli from DTCli. Similar for test class. 3. Changed
console prompt to "apex>" from
APEXCORE-340 Naming change of apex script.
Changed following (dt->apex):
1. Script name changed to apex.
2. Class name changed to ApexCli from DTCli. Similar for test class.
3. Changed console prompt to "apex>" from "dt>"
4. Changed Welcome message to "Apex CLI ..." from "DT CLI ..."
5. Updated document for name change of dtcli to apex.
6. Changing checkstyle supression for ApexCli.java
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/0b124a5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/0b124a5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/0b124a5f
Branch: refs/heads/master
Commit: 0b124a5f8e14075d2aef702ded788b5eed5e4b44
Parents: 85db448
Author: chinmaykolhatkar <ch...@datatorrent.com>
Authored: Mon Apr 11 15:56:42 2016 +0530
Committer: chinmaykolhatkar <ch...@datatorrent.com>
Committed: Wed Apr 13 09:49:05 2016 +0530
----------------------------------------------------------------------
docs/apex_cli.md | 268 ++
docs/apex_development_setup.md | 2 +-
docs/application_development.md | 20 +-
docs/application_packages.md | 10 +-
docs/dtcli.md | 268 --
docs/images/security/image02.png | Bin 11594 -> 18082 bytes
docs/security.md | 18 +-
engine/engine-checkstyle-suppressions.xml | 2 +-
.../java/com/datatorrent/stram/cli/ApexCli.java | 3950 ++++++++++++++++++
.../java/com/datatorrent/stram/cli/DTCli.java | 3950 ------------------
.../com/datatorrent/stram/cli/package-info.java | 2 +-
engine/src/main/scripts/apex | 104 +
engine/src/main/scripts/dtcli | 104 -
.../com/datatorrent/stram/cli/ApexCliTest.java | 183 +
.../com/datatorrent/stram/cli/DTCliTest.java | 180 -
mkdocs.yml | 2 +-
16 files changed, 4533 insertions(+), 4530 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/apex_cli.md
----------------------------------------------------------------------
diff --git a/docs/apex_cli.md b/docs/apex_cli.md
new file mode 100644
index 0000000..0d17645
--- /dev/null
+++ b/docs/apex_cli.md
@@ -0,0 +1,268 @@
+Apache Apex Command Line Interface
+================================================================================
+
+Apex CLI, the Apache Apex command line interface, can be used to launch, monitor, and manage Apache Apex applications. It provides a developer friendly way of interacting with Apache Apex platform. Another advantage of Apex CLI is to provide scope, by connecting and executing commands in a context of specific application. Apex CLI enables easy integration with existing enterprise toolset for automated application monitoring and management. Currently the following high level tasks are supported.
+
+- Launch or kill applications
+- View system metrics including load, throughput, latency, etc.
+- Start or stop tuple recording
+- Read operator, stream, port properties and attributes
+- Write to operator properties
+- Dynamically change the application logical plan
+- Create custom macros
+
+
+## Apex CLI Commands
+
+Apex CLI can be launched by running following command
+
+ apex
+
+Help on all commands is available via “help” command in the CLI
+
+### Global Commands
+
+```
+GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
+
+alias alias-name command
+ Create a command alias
+
+begin-macro name
+ Begin Macro Definition ($1...$9 to access parameters and type 'end' to end the definition)
+
+connect app-id
+ Connect to an app
+
+dump-properties-file out-file jar-file class-name
+ Dump the properties file of an app class
+
+echo [arg ...]
+ Echo the arguments
+
+exit
+ Exit the CLI
+
+get-app-info app-id
+ Get the information of an app
+
+get-app-package-info app-package-file
+ Get info on the app package file
+
+get-app-package-operator-properties app-package-file operator-class
+ Get operator properties within the given app package
+
+get-app-package-operators [options] app-package-file [search-term]
+ Get operators within the given app package
+ Options:
+ -parent Specify the parent class for the operators
+
+get-config-parameter [parameter-name]
+ Get the configuration parameter
+
+get-jar-operator-classes [options] jar-files-comma-separated [search-term]
+ List operators in a jar list
+ Options:
+ -parent Specify the parent class for the operators
+
+get-jar-operator-properties jar-files-comma-separated operator-class-name
+ List properties in specified operator
+
+help [command]
+ Show help
+
+kill-app app-id [app-id ...]
+ Kill an app
+
+ launch [options] jar-file/json-file/properties-file/app-package-file [matching-app-name]
+ Launch an app
+ Options:
+ -apconf <app package configuration file> Specify an application
+ configuration file
+ within the app
+ package if launching
+ an app package.
+ -archives <comma separated list of archives> Specify comma
+ separated archives
+ to be unarchived on
+ the compute machines.
+ -conf <configuration file> Specify an
+ application
+ configuration file.
+ -D <property=value> Use value for given
+ property.
+ -exactMatch Only consider
+ applications with
+ exact app name
+ -files <comma separated list of files> Specify comma
+ separated files to
+ be copied on the
+ compute machines.
+ -ignorepom Do not run maven to
+ find the dependency
+ -libjars <comma separated list of libjars> Specify comma
+ separated jar files
+ or other resource
+ files to include in
+ the classpath.
+ -local Run application in
+ local mode.
+ -originalAppId <application id> Specify original
+ application
+ identifier for restart.
+ -queue <queue name> Specify the queue to
+ launch the application
+
+list-application-attributes
+ Lists the application attributes
+list-apps [pattern]
+ List applications
+list-operator-attributes
+ Lists the operator attributes
+list-port-attributes
+ Lists the port attributes
+set-pager on/off
+ Set the pager program for output
+show-logical-plan [options] jar-file/app-package-file [class-name]
+ List apps in a jar or show logical plan of an app class
+ Options:
+ -exactMatch Only consider exact match
+ for app name
+ -ignorepom Do not run maven to find
+ the dependency
+ -libjars <comma separated list of jars> Specify comma separated
+ jar/resource files to
+ include in the classpath.
+shutdown-app app-id [app-id ...]
+ Shutdown an app
+source file
+ Execute the commands in a file
+```
+
+### Commands after connecting to an application
+
+```
+COMMANDS WHEN CONNECTED TO AN APP (via connect <appid>) EXCEPT WHEN CHANGING LOGICAL PLAN:
+
+begin-logical-plan-change
+ Begin Logical Plan Change
+dump-properties-file out-file [jar-file] [class-name]
+ Dump the properties file of an app class
+get-app-attributes [attribute-name]
+ Get attributes of the connected app
+get-app-info [app-id]
+ Get the information of an app
+get-operator-attributes operator-name [attribute-name]
+ Get attributes of an operator
+get-operator-properties operator-name [property-name]
+ Get properties of a logical operator
+get-physical-operator-properties [options] operator-id
+ Get properties of a physical operator
+ Options:
+ -propertyName <property name> The name of the property whose
+ value needs to be retrieved
+ -waitTime <wait time> How long to wait to get the result
+get-port-attributes operator-name port-name [attribute-name]
+ Get attributes of a port
+get-recording-info [operator-id] [start-time]
+ Get tuple recording info
+kill-app [app-id ...]
+ Kill an app
+kill-container container-id [container-id ...]
+ Kill a container
+list-containers
+ List containers
+list-operators [pattern]
+ List operators
+set-operator-property operator-name property-name property-value
+ Set a property of an operator
+set-physical-operator-property operator-id property-name property-value
+ Set a property of an operator
+show-logical-plan [options] [jar-file/app-package-file] [class-name]
+ Show logical plan of an app class
+ Options:
+ -exactMatch Only consider exact match
+ for app name
+ -ignorepom Do not run maven to find
+ the dependency
+ -libjars <comma separated list of jars> Specify comma separated
+ jar/resource files to
+ include in the classpath.
+show-physical-plan
+ Show physical plan
+shutdown-app [app-id ...]
+ Shutdown an app
+start-recording operator-id [port-name] [num-windows]
+ Start recording
+stop-recording operator-id [port-name]
+ Stop recording
+wait timeout
+ Wait for completion of current application
+```
+
+### Commands when changing the logical plan
+
+```
+COMMANDS WHEN CHANGING LOGICAL PLAN (via begin-logical-plan-change):
+
+abort
+ Abort the plan change
+add-stream-sink stream-name to-operator-name to-port-name
+ Add a sink to an existing stream
+create-operator operator-name class-name
+ Create an operator
+create-stream stream-name from-operator-name from-port-name to-operator-name to-port-name
+ Create a stream
+help [command]
+ Show help
+remove-operator operator-name
+ Remove an operator
+remove-stream stream-name
+ Remove a stream
+set-operator-attribute operator-name attr-name attr-value
+ Set an attribute of an operator
+set-operator-property operator-name property-name property-value
+ Set a property of an operator
+set-port-attribute operator-name port-name attr-name attr-value
+ Set an attribute of a port
+set-stream-attribute stream-name attr-name attr-value
+ Set an attribute of a stream
+show-queue
+ Show the queue of the plan change
+submit
+ Submit the plan change
+```
+
+
+
+## Examples
+
+An example of defining a custom macro. The macro updates a running application by inserting a new operator. It takes three parameters and executes a logical plan changes.
+
+```
+apex> begin-macro add-console-output
+macro> begin-logical-plan-change
+macro> create-operator $1 com.datatorrent.lib.io.ConsoleOutputOperator
+macro> create-stream stream_$1 $2 $3 $1 in
+macro> submit
+```
+
+
+Then execute the `add-console-output` macro like this
+
+```
+apex> add-console-output xyz opername portname
+```
+
+This macro then expands to run the following command
+
+```
+begin-logical-plan-change
+create-operator xyz com.datatorrent.lib.io.ConsoleOutputOperator
+create-stream stream_xyz opername portname xyz in
+submit
+```
+
+
+*Note*: To perform runtime logical plan changes, like ability to add new operators,
+they must be part of the jar files that were deployed at application launch time.
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/apex_development_setup.md
----------------------------------------------------------------------
diff --git a/docs/apex_development_setup.md b/docs/apex_development_setup.md
index d7c4d98..ce1651c 100644
--- a/docs/apex_development_setup.md
+++ b/docs/apex_development_setup.md
@@ -92,7 +92,7 @@ When the run completes successfully, you should see a new directory named `myape
cd myapexapp
mvn clean package -DskipTests
-The build should create the application package file `myapexapp/target/myapexapp-1.0-SNAPSHOT.apa`. This application package can then be used to launch example application via **dtCli**, or other visual management tools. When running, this application will generate a stream of random numbers and print them out, each prefixed by the string `hello world:`.
+The build should create the application package file `myapexapp/target/myapexapp-1.0-SNAPSHOT.apa`. This application package can then be used to launch example application via **apex** CLI, or other visual management tools. When running, this application will generate a stream of random numbers and print them out, each prefixed by the string `hello world:`.
Running Unit Tests
----
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/application_development.md
----------------------------------------------------------------------
diff --git a/docs/application_development.md b/docs/application_development.md
index a720378..8f98eba 100644
--- a/docs/application_development.md
+++ b/docs/application_development.md
@@ -612,25 +612,25 @@ local mode (in IDE or from command line) or on a Hadoop cluster.
-To start the dtCli run
+To start the Apex CLI run
- <INSTALL_DIR>/bin/dtcli
+ <INSTALL_DIR>/bin/apex
The command line prompt appears. To start the application in local mode (the actual version number in the file name may differ)
- dt> launch -local <INSTALL_DIR>/yahoo-finance-demo-3.2.0-SNAPSHOT.apa
+ apex> launch -local <INSTALL_DIR>/yahoo-finance-demo-3.2.0-SNAPSHOT.apa
To terminate the application in local mode, enter Ctrl-C
Tu run the application on the Hadoop cluster (the actual version
number in the file name may differ)
- dt> launch <INSTALL_DIR>/yahoo-finance-demo-3.2.0-SNAPSHOT.apa
+ apex> launch <INSTALL_DIR>/yahoo-finance-demo-3.2.0-SNAPSHOT.apa
-To stop the application running in Hadoop, terminate it in the dtCli:
+To stop the application running in Hadoop, terminate it in the Apex CLI:
- dt> kill-app
+ apex> kill-app
@@ -1079,7 +1079,7 @@ libraries, version of the platform etc.)
#### Running an application
-The platform provides a commandline tool called dtcli for managing applications (launching,
+The platform provides a commandline tool called *apex* for managing applications (launching,
killing, viewing, etc.). This tool was already discussed above briefly
in the section entitled Running the Test Application. It will introspect
the jar file specified with the launch command for applications (classes
@@ -1089,7 +1089,7 @@ application package to the cluster.
-Dtcli can run the application in local mode (i.e. outside a
+Apex CLI can run the application in local mode (i.e. outside a
cluster). It is recommended to first run the application in local mode
in the development environment before launching on the Hadoop cluster.
This way some of the external system integration and correct
@@ -1098,7 +1098,7 @@ environment before testing distributed mode.
-For more details on CLI please refer to the [dtCli Guide](dtcli.md).
+For more details on CLI please refer to the [Apex CLI Guide](apex_cli.md).
Application API
----------------------------
@@ -1332,7 +1332,7 @@ an id. This id along with the Hadoop container name uniquely identifies
the operator in the execution plan of the DAG. The logical names and the
physical names are required for web service support. Operators can be
accessed via both names. These same names are used while interacting
-with dtcli to access an operator.
+with apex cli to access an operator.
Ideally these names should be self-descriptive. For example in Figure 1,
the node named “Daily volume” has a physical identifier of 2.
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/application_packages.md
----------------------------------------------------------------------
diff --git a/docs/application_packages.md b/docs/application_packages.md
index 06f5696..67cfa19 100644
--- a/docs/application_packages.md
+++ b/docs/application_packages.md
@@ -484,7 +484,7 @@ There are four top level directories in an Application Package:
## Examining and Launching Application Packages Through CLI
-If you are working with Application Packages in the local filesystem, you can use the Apex Command Line Interface (dtcli).
+If you are working with Application Packages in the local filesystem, you can use the Apex Command Line Interface (apex).
### Getting Application Package Meta Information
@@ -492,7 +492,7 @@ You can get the meta information about the Application Package using
this Apex CLI command.
```
- dt> get-app-package-info <app-package-file>
+ apex> get-app-package-info <app-package-file>
```
### Getting Available Operators In Application Package
@@ -501,7 +501,7 @@ You can get the list of available operators in the Application Package
using this command.
```
- dt> get-app-package-operators <app-package-file> <package-prefix>
+ apex> get-app-package-operators <app-package-file> <package-prefix>
[parent-class]
```
@@ -510,14 +510,14 @@ using this command.
You can get the list of properties of any operator in the Application
Package using this command.
- dt> get-app-package-operator-properties <app-package-file> <operator-class>
+ apex> get-app-package-operator-properties <app-package-file> <operator-class>
### Launching an Application Package
You can launch an application within an Application Package.
```
-dt> launch [-D property-name=property-value, ...] [-conf config-name]
+apex> launch [-D property-name=property-value, ...] [-conf config-name]
[-apconf config-file-within-app-package] <app-package-file>
[matching-app-name]
```
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/dtcli.md
----------------------------------------------------------------------
diff --git a/docs/dtcli.md b/docs/dtcli.md
deleted file mode 100644
index 8cf11e6..0000000
--- a/docs/dtcli.md
+++ /dev/null
@@ -1,268 +0,0 @@
-Apache Apex Command Line Interface
-================================================================================
-
-dtCli, the Apache Apex command line interface, can be used to launch, monitor, and manage Apache Apex applications. It provides a developer friendly way of interacting with Apache Apex platform. Another advantage of dtCli is to provide scope, by connecting and executing commands in a context of specific application. dtCli enables easy integration with existing enterprise toolset for automated application monitoring and management. Currently the following high level tasks are supported.
-
-- Launch or kill applications
-- View system metrics including load, throughput, latency, etc.
-- Start or stop tuple recording
-- Read operator, stream, port properties and attributes
-- Write to operator properties
-- Dynamically change the application logical plan
-- Create custom macros
-
-
-## dtcli Commands
-
-dtCli can be launched by running following command
-
- dtcli
-
-Help on all commands is available via “help” command in the CLI
-
-### Global Commands
-
-```
-GLOBAL COMMANDS EXCEPT WHEN CHANGING LOGICAL PLAN:
-
-alias alias-name command
- Create a command alias
-
-begin-macro name
- Begin Macro Definition ($1...$9 to access parameters and type 'end' to end the definition)
-
-connect app-id
- Connect to an app
-
-dump-properties-file out-file jar-file class-name
- Dump the properties file of an app class
-
-echo [arg ...]
- Echo the arguments
-
-exit
- Exit the CLI
-
-get-app-info app-id
- Get the information of an app
-
-get-app-package-info app-package-file
- Get info on the app package file
-
-get-app-package-operator-properties app-package-file operator-class
- Get operator properties within the given app package
-
-get-app-package-operators [options] app-package-file [search-term]
- Get operators within the given app package
- Options:
- -parent Specify the parent class for the operators
-
-get-config-parameter [parameter-name]
- Get the configuration parameter
-
-get-jar-operator-classes [options] jar-files-comma-separated [search-term]
- List operators in a jar list
- Options:
- -parent Specify the parent class for the operators
-
-get-jar-operator-properties jar-files-comma-separated operator-class-name
- List properties in specified operator
-
-help [command]
- Show help
-
-kill-app app-id [app-id ...]
- Kill an app
-
- launch [options] jar-file/json-file/properties-file/app-package-file [matching-app-name]
- Launch an app
- Options:
- -apconf <app package configuration file> Specify an application
- configuration file
- within the app
- package if launching
- an app package.
- -archives <comma separated list of archives> Specify comma
- separated archives
- to be unarchived on
- the compute machines.
- -conf <configuration file> Specify an
- application
- configuration file.
- -D <property=value> Use value for given
- property.
- -exactMatch Only consider
- applications with
- exact app name
- -files <comma separated list of files> Specify comma
- separated files to
- be copied on the
- compute machines.
- -ignorepom Do not run maven to
- find the dependency
- -libjars <comma separated list of libjars> Specify comma
- separated jar files
- or other resource
- files to include in
- the classpath.
- -local Run application in
- local mode.
- -originalAppId <application id> Specify original
- application
- identifier for restart.
- -queue <queue name> Specify the queue to
- launch the application
-
-list-application-attributes
- Lists the application attributes
-list-apps [pattern]
- List applications
-list-operator-attributes
- Lists the operator attributes
-list-port-attributes
- Lists the port attributes
-set-pager on/off
- Set the pager program for output
-show-logical-plan [options] jar-file/app-package-file [class-name]
- List apps in a jar or show logical plan of an app class
- Options:
- -exactMatch Only consider exact match
- for app name
- -ignorepom Do not run maven to find
- the dependency
- -libjars <comma separated list of jars> Specify comma separated
- jar/resource files to
- include in the classpath.
-shutdown-app app-id [app-id ...]
- Shutdown an app
-source file
- Execute the commands in a file
-```
-
-### Commands after connecting to an application
-
-```
-COMMANDS WHEN CONNECTED TO AN APP (via connect <appid>) EXCEPT WHEN CHANGING LOGICAL PLAN:
-
-begin-logical-plan-change
- Begin Logical Plan Change
-dump-properties-file out-file [jar-file] [class-name]
- Dump the properties file of an app class
-get-app-attributes [attribute-name]
- Get attributes of the connected app
-get-app-info [app-id]
- Get the information of an app
-get-operator-attributes operator-name [attribute-name]
- Get attributes of an operator
-get-operator-properties operator-name [property-name]
- Get properties of a logical operator
-get-physical-operator-properties [options] operator-id
- Get properties of a physical operator
- Options:
- -propertyName <property name> The name of the property whose
- value needs to be retrieved
- -waitTime <wait time> How long to wait to get the result
-get-port-attributes operator-name port-name [attribute-name]
- Get attributes of a port
-get-recording-info [operator-id] [start-time]
- Get tuple recording info
-kill-app [app-id ...]
- Kill an app
-kill-container container-id [container-id ...]
- Kill a container
-list-containers
- List containers
-list-operators [pattern]
- List operators
-set-operator-property operator-name property-name property-value
- Set a property of an operator
-set-physical-operator-property operator-id property-name property-value
- Set a property of an operator
-show-logical-plan [options] [jar-file/app-package-file] [class-name]
- Show logical plan of an app class
- Options:
- -exactMatch Only consider exact match
- for app name
- -ignorepom Do not run maven to find
- the dependency
- -libjars <comma separated list of jars> Specify comma separated
- jar/resource files to
- include in the classpath.
-show-physical-plan
- Show physical plan
-shutdown-app [app-id ...]
- Shutdown an app
-start-recording operator-id [port-name] [num-windows]
- Start recording
-stop-recording operator-id [port-name]
- Stop recording
-wait timeout
- Wait for completion of current application
-```
-
-### Commands when changing the logical plan
-
-```
-COMMANDS WHEN CHANGING LOGICAL PLAN (via begin-logical-plan-change):
-
-abort
- Abort the plan change
-add-stream-sink stream-name to-operator-name to-port-name
- Add a sink to an existing stream
-create-operator operator-name class-name
- Create an operator
-create-stream stream-name from-operator-name from-port-name to-operator-name to-port-name
- Create a stream
-help [command]
- Show help
-remove-operator operator-name
- Remove an operator
-remove-stream stream-name
- Remove a stream
-set-operator-attribute operator-name attr-name attr-value
- Set an attribute of an operator
-set-operator-property operator-name property-name property-value
- Set a property of an operator
-set-port-attribute operator-name port-name attr-name attr-value
- Set an attribute of a port
-set-stream-attribute stream-name attr-name attr-value
- Set an attribute of a stream
-show-queue
- Show the queue of the plan change
-submit
- Submit the plan change
-```
-
-
-
-## Examples
-
-An example of defining a custom macro. The macro updates a running application by inserting a new operator. It takes three parameters and executes a logical plan changes.
-
-```
-dt> begin-macro add-console-output
-macro> begin-logical-plan-change
-macro> create-operator $1 com.datatorrent.lib.io.ConsoleOutputOperator
-macro> create-stream stream_$1 $2 $3 $1 in
-macro> submit
-```
-
-
-Then execute the `add-console-output` macro like this
-
-```
-dt> add-console-output xyz opername portname
-```
-
-This macro then expands to run the following command
-
-```
-begin-logical-plan-change
-create-operator xyz com.datatorrent.lib.io.ConsoleOutputOperator
-create-stream stream_xyz opername portname xyz in
-submit
-```
-
-
-*Note*: To perform runtime logical plan changes, like ability to add new operators,
-they must be part of the jar files that were deployed at application launch time.
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/images/security/image02.png
----------------------------------------------------------------------
diff --git a/docs/images/security/image02.png b/docs/images/security/image02.png
index 65d30a8..e16f2ec 100644
Binary files a/docs/images/security/image02.png and b/docs/images/security/image02.png differ
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/docs/security.md
----------------------------------------------------------------------
diff --git a/docs/security.md b/docs/security.md
index af157ce..970b507 100644
--- a/docs/security.md
+++ b/docs/security.md
@@ -41,7 +41,7 @@ hdfs-site.xml and can be specified as follows for example for a lifetime of 1 ye
###CLI Configuration
The Apex command line interface is used to launch
-applications along with performing various other operations and administrative tasks on the applications. When Kerberos security is enabled in Hadoop, a Kerberos ticket granting ticket (TGT) or the Kerberos credentials of the user are needed by the CLI program `dtcli` to authenticate with Hadoop for any operation. Kerberos credentials are composed of a principal and either a _keytab_ or a password. For security and operational reasons only keytabs are supported in Hadoop and by extension in Apex platform. When user credentials are specified, all operations including launching
+applications along with performing various other operations and administrative tasks on the applications. When Kerberos security is enabled in Hadoop, a Kerberos ticket granting ticket (TGT) or the Kerberos credentials of the user are needed by the CLI program `apex` to authenticate with Hadoop for any operation. Kerberos credentials are composed of a principal and either a _keytab_ or a password. For security and operational reasons only keytabs are supported in Hadoop and by extension in Apex platform. When user credentials are specified, all operations including launching
application are performed as that user.
#### Using kinit
@@ -50,11 +50,11 @@ A Kerberos ticket granting ticket (TGT) can be obtained by using the Kerberos co
kinit -k -t path-tokeytab-file kerberos-principal
-If this command is successful, the TGT is obtained, cached and available for other programs. The CLI program `dtcli` can then be started to launch applications and perform other operations.
+If this command is successful, the TGT is obtained, cached and available for other programs. The CLI program `apex` can then be started to launch applications and perform other operations.
#### Using Kerberos credentials
-The CLI program `dtcli` can also use the Kerberos credentials directly without requiring a TGT to be obtained separately. This can be useful in batch mode where `dtcli` is not launched manually and also in scenarios where running another program like `kinit` is not feasible.
+The CLI program `apex` can also use the Kerberos credentials directly without requiring a TGT to be obtained separately. This can be useful in batch mode where `apex` is not launched manually and also in scenarios where running another program like `kinit` is not feasible.
The credentials can be specified in the `dt-site.xml` configuration file. If only a single user is launching applications, the global `dt-site.xml` configuration file in the installation folder can be used. In a multi-user environment the users can use the `dt-site.xml` file in their
home directory. The location of this file will be `$HOME/.dt/dt-site.xml`. If this file does not exist, the user can create a new one.
@@ -83,14 +83,14 @@ In this section we will see how security works for applications built on Apex. W
###Application Launch
-To launch applications in Apache Apex the command line client dtcli can be used. The application artifacts such as binaries and properties are supplied as an application package. The client, during the various steps involved to launch the application needs to communicate with both the Resource Manager and the Name Node. The Resource Manager communication involves the client asking for new resources to run the application master and start the application launch process. The steps along with sample Java code are described in Writing YARN Applications. The Name Node communication includes the application artifacts being copied to HDFS so that they are available across the cluster for launching the different application containers.
+To launch applications in Apache Apex the command line client `apex` can be used. The application artifacts such as binaries and properties are supplied as an application package. The client, during the various steps involved to launch the application needs to communicate with both the Resource Manager and the Name Node. The Resource Manager communication involves the client asking for new resources to run the application master and start the application launch process. The steps along with sample Java code are described in Writing YARN Applications. The Name Node communication includes the application artifacts being copied to HDFS so that they are available across the cluster for launching the different application containers.
In secure mode, the communications with both Resource Manager and Name Node requires authentication and the mechanism is Kerberos. Below is an illustration showing this.
-
+
-The client dtcli supports Kerberos authentication and will automatically enable it in a secure environment. To authenticate, some Kerberos configuration namely the Kerberos credentials, are needed by the client. There are two parameters, the Kerberos principal and keytab to use for the client. These can be specified in the dt-site.xml configuration file. The properties are shown below
+The client `apex` supports Kerberos authentication and will automatically enable it in a secure environment. To authenticate, some Kerberos configuration namely the Kerberos credentials, are needed by the client. There are two parameters, the Kerberos principal and keytab to use for the client. These can be specified in the dt-site.xml configuration file. The properties are shown below
<property>
<name>dt.authentication.principal</name>
@@ -116,13 +116,13 @@ When the application is completely up and running, there are different component
Every Apache Apex application has a master process akin to any YARN application. In our case it is called STRAM (Streaming Application Master). It is a master process that runs in its own container and manages the different distributed components of the application. Among other tasks it requests Resource Manager for new resources as they are needed and gives back resources that are no longer needed. STRAM also needs to communicate with Name Node from time-to-time to access the persistent HDFS file system.
-In secure mode, STRAM has to authenticate with both Resource Manager and Name Node before it can send any requests and this is achieved using Delegation Tokens. Since STRAM runs as a managed application master, it runs in a Hadoop container. This container could have been allocated on any node based on what resources were available. Since there is no fixed node where STRAM runs, it does not have Kerberos credentials. Unlike launch client dtcli, it cannot authenticate with Hadoop services Resource Manager and Name Node using Kerberos. Instead, Delegation Tokens are used for authentication.
+In secure mode, STRAM has to authenticate with both Resource Manager and Name Node before it can send any requests and this is achieved using Delegation Tokens. Since STRAM runs as a managed application master, it runs in a Hadoop container. This container could have been allocated on any node based on what resources were available. Since there is no fixed node where STRAM runs, it does not have Kerberos credentials. Unlike launch client `apex`, it cannot authenticate with Hadoop services Resource Manager and Name Node using Kerberos. Instead, Delegation Tokens are used for authentication.
#####Delegation Tokens
-Delegation tokens are tokens that are dynamically issued by the source and clients use them to authenticate with the source. The source stores the delegation tokens it has issued in a cache and checks the delegation token sent by a client against the cache. If a match is found, the authentication is successful else it fails. This is the second mode of authentication in secure Hadoop after Kerberos. More details can be found in the Hadoop security design document. In this case the delegation tokens are issued by Resource Manager and Name Node. STRAM would use these tokens to authenticate with them. But how does it get them in the first place? This is where the launch client dtcli comes in.
+Delegation tokens are tokens that are dynamically issued by the source and clients use them to authenticate with the source. The source stores the delegation tokens it has issued in a cache and checks the delegation token sent by a client against the cache. If a match is found, the authentication is successful else it fails. This is the second mode of authentication in secure Hadoop after Kerberos. More details can be found in the Hadoop security design document. In this case the delegation tokens are issued by Resource Manager and Name Node. STRAM would use these tokens to authenticate with them. But how does it get them in the first place? This is where the launch client `apex` comes in.
-The client dtcli, since it possesses Kerberos credentials as explained in the Application Launch section, is able to authenticate with Resource Manager and Name Node using Kerberos. It then requests for delegation tokens over the Kerberos authenticated connection. The servers return the delegation tokens in the response payload. The client in requesting the resource manager for the start of the application master container for STRAM seeds it with these tokens so that when STRAM starts it has these tokens. It can then use these tokens to authenticate with the Hadoop services.
+The client `apex`, since it possesses Kerberos credentials as explained in the Application Launch section, is able to authenticate with Resource Manager and Name Node using Kerberos. It then requests for delegation tokens over the Kerberos authenticated connection. The servers return the delegation tokens in the response payload. The client in requesting the resource manager for the start of the application master container for STRAM seeds it with these tokens so that when STRAM starts it has these tokens. It can then use these tokens to authenticate with the Hadoop services.
####Streaming Container
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/0b124a5f/engine/engine-checkstyle-suppressions.xml
----------------------------------------------------------------------
diff --git a/engine/engine-checkstyle-suppressions.xml b/engine/engine-checkstyle-suppressions.xml
index 9614eef..9334631 100644
--- a/engine/engine-checkstyle-suppressions.xml
+++ b/engine/engine-checkstyle-suppressions.xml
@@ -24,7 +24,7 @@
"http://www.puppycrawl.com/dtds/suppressions_1_0.dtd">
<suppressions>
- <suppress checks="RegexpMultiline" files="DTCli.java"/>
+ <suppress checks="RegexpMultiline" files="ApexCli.java"/>
<suppress checks="RegexpMultiline" files=".*Tests?.java"/>
<suppress checks="RegexpMultiline" files="VersionInfo.java"/>
<suppress checks="NoFinalizer" files="FSRecoveryHandler.java"/>