You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@apex.apache.org by th...@apache.org on 2016/03/03 00:32:51 UTC
[22/50] incubator-apex-core git commit: Fixed formatting and image
Fixed formatting and image
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/6b79095c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/6b79095c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/6b79095c
Branch: refs/heads/APEXCORE-293
Commit: 6b79095cf6547da41bc55d7ea2cacc4e18899c17
Parents: d373226
Author: Chandni Singh <cs...@apache.org>
Authored: Wed Nov 4 18:53:00 2015 -0800
Committer: Thomas Weise <th...@datatorrent.com>
Committed: Sun Feb 28 22:46:36 2016 -0800
----------------------------------------------------------------------
autometrics/autometrics.md | 52 ++++++++++++++++++++++++----------------
autometrics/images/adt.png | Bin 0 -> 169889 bytes
2 files changed, 31 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/6b79095c/autometrics/autometrics.md
----------------------------------------------------------------------
diff --git a/autometrics/autometrics.md b/autometrics/autometrics.md
index 8e7ffb2..364007f 100644
--- a/autometrics/autometrics.md
+++ b/autometrics/autometrics.md
@@ -51,12 +51,13 @@ For the `LineReceiver` operator, the application developer need not specify any
## Building custom aggregators
-Platform cannot perform any meaningful aggregation for a non-numeric metric. In such case, the operator or application developer can write custom aggregators. Let’s say, in the
+Platform cannot perform any meaningful aggregation for a non-numeric metric. In such case, the operator or application developer can write custom aggregators. Let’s say, if the `LineReceiver` was modified to
-Retrieving AutoMetrics
+# Retrieving AutoMetrics
The Gateway REST API provides a way to retrieve the latest AutoMetrics for each logical operator. For example:
+```
GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName}
{
...
@@ -65,20 +66,23 @@ GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName}
}
...
}
+```
-System Metrics
+# System Metrics
System metrics are standard operator metrics provided by the system. Examples include:
-processed tuples per second
-emitted tuples per second
-total tuples processed
-total tuples emitted
-latency
-CPU percentage
-failure count
-checkpoint elapsed time
+- processed tuples per second
+- emitted tuples per second
+- total tuples processed
+- total tuples emitted
+- latency
+- CPU percentage
+- failure count
+- checkpoint elapsed time
The Gateway REST API provides a way to retrieve the latest values for all of the above for each of the logical operators in the application.
+
+```
GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName}
{
...
@@ -91,17 +95,18 @@ GET /ws/v2/applications/{appid}/logicalPlan/operators/{opName}
"tuplesProcessedPSMA": "{tuplesProcessedPSMA}",
...
}
+```
However, just like AutoMetrics, the Gateway only provides the latest metrics. For historical metrics, we will need the help of App Data Tracker.
-App Data Tracker
+# App Data Tracker
As discussed above, STRAM aggregates the AutoMetrics from physical operators (partitions) to something that makes sense in one logical operator. It pushes the aggregated AutoMetrics values using Websocket to the Gateway at every second along with system metrics for each operator. Gateway relays the information to an application called App Data Tracker. It is another Apex application that runs in the background and further aggregates the incoming values by time bucket and stores the values in HDHT. It also allows the outside to retrieve the aggregated AutoMetrics and system metrics through websocket interface.
-
-
+
App Data Tracker is enabled by having these properties in dt-site.xml:
+```xml
<property>
<name>dt.appDataTracker.enable</name>
<value>true</value>
@@ -114,19 +119,21 @@ App Data Tracker is enabled by having these properties in dt-site.xml:
<name>dt.attr.METRICS_TRANSPORT</name>
<value>builtin:AppDataTrackerFeed</value>
</property>
+```
All the applications launched after the App Data Tracker is enabled will have metrics sent to it.
-Note: The App Data Tracker will be shown running in dtManage as a “system app”. It will show up if the “show system apps” button is pressed.
+**Note**: The App Data Tracker will be shown running in dtManage as a “system app”. It will show up if the “show system apps” button is pressed.
+
+By default, the time buckets App Data Tracker aggregates upon are one minute, one hour and one day. It can be overridden by changing the operator attribute `METRICS_DIMENSIONS_SCHEME`.
-By default, the time buckets App Data Tracker aggregates upon are one minute, one hour and one day. It can be overridden by changing the operator attribute METRICS_DIMENSIONS_SCHEME.
+Also by default, the app data tracker performs all these aggregations: SUM, MIN, MAX, AVG, COUNT, FIRST, LAST on all number metrics. You can also override by changing the same operator attribute `METRICS_DIMENSIONS_SCHEME`, provided the custom aggregator is known to the App Data Tracker. (See next section)
-Also by default, the app data tracker performs all these aggregations: SUM, MIN, MAX, AVG, COUNT, FIRST, LAST on all number metrics. You can also override by changing the same operator attribute METRICS_DIMENSIONS_SCHEME, provided the custom aggregator is known to the App Data Tracker. (See next section)
-Custom Aggregator in App Data Tracker
+# Custom Aggregator in App Data Tracker
Custom aggregators allow you to do your own custom computation on statistics generated by any of your applications. In order to implement a Custom aggregator you have to do two things:
-Combining new inputs with the current aggregation
-Combining two aggregations together into one aggregation
+1. Combining new inputs with the current aggregation
+2. Combining two aggregations together into one aggregation
Let’s consider the case where we want to perform the following rolling average:
@@ -134,6 +141,7 @@ Y_n = ½ * X_n + ½ * X_n-1 + ¼ * X_n-2 + ⅛ * X_n-3 +...
This aggregation could be performed by the following Custom Aggregator:
+```java
@Name("IIRAVG")
public class AggregatorIIRAVG extends AbstractIncrementalAggregator
{
@@ -163,7 +171,9 @@ public class AggregatorIIRAVG extends AbstractIncrementalAggregator
aggregateHelper(destAgg, srcAgg);
}
}
-Dashboards
+```
+
+# Dashboards
With App Data Tracker enabled, you can visualize the AutoMetrics and system metrics in the Dashboards within dtManage. Refer back to the diagram in the App Data Tracker section, dtGateway relays queries and query results to and from the App Data Tracker. In this way, dtManage sends queries and receives results from the App Data Tracker via dtGateway and uses the results to let the user visualize the data.
The Dashboards have some simple builtin widgets to visualize the data. Line charts and bar charts are some examples.
http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/6b79095c/autometrics/images/adt.png
----------------------------------------------------------------------
diff --git a/autometrics/images/adt.png b/autometrics/images/adt.png
new file mode 100644
index 0000000..5b0d8ea
Binary files /dev/null and b/autometrics/images/adt.png differ