You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bigtop.apache.org by kw...@apache.org on 2017/04/25 20:42:51 UTC
[2/2] bigtop git commit: BIGTOP-2747: new charm revs for bigtop-1.2
(closes #197)
BIGTOP-2747: new charm revs for bigtop-1.2 (closes #197)
Signed-off-by: Kevin W Monroe <ke...@canonical.com>
Project: http://git-wip-us.apache.org/repos/asf/bigtop/repo
Commit: http://git-wip-us.apache.org/repos/asf/bigtop/commit/b5b366fe
Tree: http://git-wip-us.apache.org/repos/asf/bigtop/tree/b5b366fe
Diff: http://git-wip-us.apache.org/repos/asf/bigtop/diff/b5b366fe
Branch: refs/heads/master
Commit: b5b366fe1e535b3549d3b1c563bdce65b0d21914
Parents: 2f8311b
Author: Kevin W Monroe <ke...@canonical.com>
Authored: Wed Apr 19 17:31:52 2017 +0000
Committer: Kevin W Monroe <ke...@canonical.com>
Committed: Tue Apr 25 15:42:34 2017 -0500
----------------------------------------------------------------------
bigtop-deploy/juju/hadoop-hbase/README.md | 67 +++++-----
bigtop-deploy/juju/hadoop-hbase/bundle.yaml | 12 +-
bigtop-deploy/juju/hadoop-kafka/README.md | 49 +++----
bigtop-deploy/juju/hadoop-kafka/bundle.yaml | 12 +-
bigtop-deploy/juju/hadoop-processing/README.md | 45 +++----
.../juju/hadoop-processing/bundle.yaml | 8 +-
bigtop-deploy/juju/hadoop-spark/README.md | 134 +++++++------------
bigtop-deploy/juju/hadoop-spark/bundle.yaml | 42 ++----
.../juju/hadoop-spark/tests/tests.yaml | 8 +-
bigtop-deploy/juju/spark-processing/README.md | 109 ++++++---------
bigtop-deploy/juju/spark-processing/bundle.yaml | 4 +-
.../juju/spark-processing/tests/tests.yaml | 8 +-
bigtop-packages/src/charm/README.md | 23 ++--
.../hadoop/layer-hadoop-namenode/README.md | 39 +++---
.../hadoop/layer-hadoop-namenode/metadata.yaml | 5 +-
.../charm/hadoop/layer-hadoop-plugin/README.md | 35 +++--
.../layer-hadoop-resourcemanager/README.md | 39 +++---
.../actions/mrbench | 1 +
.../actions/nnbench | 1 +
.../actions/parseBenchmark.py | 3 +-
.../actions/smoke-test | 10 +-
.../actions/teragen | 1 +
.../actions/terasort | 1 +
.../actions/testdfsio | 1 +
.../layer-hadoop-resourcemanager/metadata.yaml | 5 +-
.../charm/hadoop/layer-hadoop-slave/README.md | 40 +++---
.../layer-hadoop-slave/actions/smoke-test | 10 +-
.../hadoop/layer-hadoop-slave/metadata.yaml | 4 +-
.../src/charm/hbase/layer-hbase/README.md | 80 ++++-------
.../charm/hbase/layer-hbase/actions/perf-test | 3 +-
.../charm/hbase/layer-hbase/actions/smoke-test | 4 +-
.../src/charm/hbase/layer-hbase/metadata.yaml | 6 +-
.../src/charm/kafka/layer-kafka/README.md | 50 +++----
.../kafka/layer-kafka/actions/create-topic | 7 +-
.../kafka/layer-kafka/actions/kafkautils.py | 4 +-
.../charm/kafka/layer-kafka/actions/list-topics | 7 +-
.../charm/kafka/layer-kafka/actions/list-zks | 9 +-
.../charm/kafka/layer-kafka/actions/read-topic | 7 +-
.../charm/kafka/layer-kafka/actions/smoke-test | 9 +-
.../charm/kafka/layer-kafka/actions/write-topic | 7 +-
.../src/charm/kafka/layer-kafka/metadata.yaml | 25 +---
.../src/charm/mahout/layer-mahout/README.md | 54 ++++----
.../mahout/layer-mahout/actions/smoke-test | 1 +
.../src/charm/mahout/layer-mahout/metadata.yaml | 7 +-
.../src/charm/pig/layer-pig/README.md | 98 +++++++-------
.../src/charm/pig/layer-pig/actions/smoke-test | 15 +--
.../src/charm/pig/layer-pig/metadata.yaml | 5 +-
.../src/charm/spark/layer-spark/README.md | 123 ++++++-----------
.../charm/spark/layer-spark/actions/pagerank | 32 ++---
.../actions/restart-spark-job-history-server | 26 ++--
.../charm/spark/layer-spark/actions/sparkbench | 10 +-
.../src/charm/spark/layer-spark/actions/sparkpi | 30 +++--
.../actions/start-spark-job-history-server | 26 ++--
.../actions/stop-spark-job-history-server | 26 ++--
.../src/charm/spark/layer-spark/actions/submit | 3 +-
.../src/charm/spark/layer-spark/metadata.yaml | 6 +-
.../src/charm/zeppelin/layer-zeppelin/README.md | 47 +++----
.../zeppelin/layer-zeppelin/actions/restart | 7 +-
.../zeppelin/layer-zeppelin/actions/smoke-test | 14 +-
.../lib/charms/layer/bigtop_zeppelin.py | 64 ++++++++-
.../charm/zeppelin/layer-zeppelin/metadata.yaml | 7 +-
.../layer-zeppelin/reactive/zeppelin.py | 3 +-
.../charm/zookeeper/layer-zookeeper/README.md | 50 +++----
.../zookeeper/layer-zookeeper/actions/restart | 11 +-
.../layer-zookeeper/actions/smoke-test | 10 +-
.../zookeeper/layer-zookeeper/metadata.yaml | 7 +-
66 files changed, 732 insertions(+), 884 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-hbase/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-hbase/README.md b/bigtop-deploy/juju/hadoop-hbase/README.md
index b45bf7b..2825327 100644
--- a/bigtop-deploy/juju/hadoop-hbase/README.md
+++ b/bigtop-deploy/juju/hadoop-hbase/README.md
@@ -26,14 +26,20 @@ to deliver high-availability, Hadoop can detect and handle failures at the
application layer. This provides a highly-available service on top of a cluster
of machines, each of which may be prone to failure.
-HBase is the Hadoop database. Think of it as a distributed, scalable Big Data
-store.
+Apache HBase is the Hadoop database. Think of it as a distributed, scalable
+Big Data store.
-This bundle provides a complete deployment of Hadoop and HBase components from
-[Apache Bigtop][] that performs distributed data processing at scale. Ganglia
-and rsyslog applications are also provided to monitor cluster health and syslog
-activity.
+Use HBase when you need random, realtime read/write access to your Big Data.
+This project's goal is the hosting of very large tables -- billions of rows X
+millions of columns -- atop clusters of commodity hardware. Learn more at
+[hbase.apache.org][].
+This bundle provides a complete deployment of Hadoop and HBase components
+from [Apache Bigtop][] that performs distributed data processing at scale.
+Ganglia and rsyslog applications are also provided to monitor cluster health
+and syslog activity.
+
+[hbase.apache.org]: http://hbase.apache.org/
[Apache Bigtop]: http://bigtop.apache.org/
## Bundle Composition
@@ -41,14 +47,14 @@ activity.
The applications that comprise this bundle are spread across 8 units as
follows:
- * NameNode (HDFS)
- * ResourceManager (YARN)
+ * NameNode v2.7.3
+ * ResourceManager v2.7.3
* Colocated on the NameNode unit
- * Zookeeper
+ * Zookeeper v3.4.6
* 3 separate units
- * Slave (DataNode and NodeManager)
+ * Slave (DataNode and NodeManager) v2.7.3
* 3 separate units
- * HBase
+ * HBase v1.1.9
* 3 units colocated with the Hadoop Slaves
* Client (Hadoop endpoint)
* Plugin (Facilitates communication with the Hadoop cluster)
@@ -65,9 +71,8 @@ demands.
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-bundle.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this bundle.
> **Note**: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
@@ -79,18 +84,10 @@ Deploy this bundle from the Juju charm store with the `juju deploy` command:
juju deploy hadoop-hbase
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-hbase`.
-
Alternatively, deploy a locally modified `bundle.yaml` with:
juju deploy /path/to/bundle.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-/path/to/bundle.yaml`.
-
The charms in this bundle can also be built from their source layers in the
[Bigtop charm repository][]. See the [Bigtop charm README][] for instructions
on building and deploying these charms locally.
@@ -102,7 +99,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/hadoop-hbase/bundle.yaml
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -138,25 +134,16 @@ complete. Run the smoke-test actions as follows:
juju run-action hbase/0 smoke-test
juju run-action zookeeper/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do <application>/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, all of the actions should settle to `status: completed`. If
any report `status: failed`, that application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
Applications in this bundle include command line and web utilities that
can be used to verify information about the cluster.
@@ -314,6 +301,18 @@ Multiple units may be added at once. For example, add four more slave units:
juju add-unit -n4 slave
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this bundle at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, bundle issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -324,6 +323,6 @@ Multiple units may be added at once. For example, add four more slave units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-hbase/bundle.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-hbase/bundle.yaml b/bigtop-deploy/juju/hadoop-hbase/bundle.yaml
index 9e3c823..618b196 100644
--- a/bigtop-deploy/juju/hadoop-hbase/bundle.yaml
+++ b/bigtop-deploy/juju/hadoop-hbase/bundle.yaml
@@ -1,6 +1,6 @@
services:
namenode:
- charm: "cs:xenial/hadoop-namenode-13"
+ charm: "cs:xenial/hadoop-namenode-14"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -9,7 +9,7 @@ services:
to:
- "0"
resourcemanager:
- charm: "cs:xenial/hadoop-resourcemanager-14"
+ charm: "cs:xenial/hadoop-resourcemanager-16"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -18,7 +18,7 @@ services:
to:
- "0"
slave:
- charm: "cs:xenial/hadoop-slave-13"
+ charm: "cs:xenial/hadoop-slave-15"
constraints: "mem=7G root-disk=32G"
num_units: 3
annotations:
@@ -29,7 +29,7 @@ services:
- "2"
- "3"
plugin:
- charm: "cs:xenial/hadoop-plugin-13"
+ charm: "cs:xenial/hadoop-plugin-14"
annotations:
gui-x: "1000"
gui-y: "400"
@@ -43,7 +43,7 @@ services:
to:
- "4"
hbase:
- charm: "cs:xenial/hbase-11"
+ charm: "cs:xenial/hbase-14"
constraints: "mem=7G root-disk=32G"
num_units: 3
annotations:
@@ -54,7 +54,7 @@ services:
- "2"
- "3"
zookeeper:
- charm: "cs:xenial/zookeeper-17"
+ charm: "cs:xenial/zookeeper-19"
constraints: "mem=3G root-disk=32G"
num_units: 3
annotations:
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-kafka/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-kafka/README.md b/bigtop-deploy/juju/hadoop-kafka/README.md
index 4c7a581..4caf774 100644
--- a/bigtop-deploy/juju/hadoop-kafka/README.md
+++ b/bigtop-deploy/juju/hadoop-kafka/README.md
@@ -44,15 +44,15 @@ activity.
The applications that comprise this bundle are spread across 9 units as
follows:
- * NameNode (HDFS)
- * ResourceManager (YARN)
+ * NameNode v2.7.3
+ * ResourceManager v2.7.3
* Colocated on the NameNode unit
- * Slave (DataNode and NodeManager)
+ * Slave (DataNode and NodeManager) v2.7.3
* 3 separate units
- * Kafka
+ * Kafka v0.10.1
* Flume-Kafka
* Colocated on the Kafka unit
- * Zookeeper
+ * Zookeeper v3.4.6
* 3 separate units
* Client (Hadoop endpoint)
* Plugin (Facilitates communication with the Hadoop cluster)
@@ -76,9 +76,8 @@ demands.
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-bundle.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this bundle.
> **Note**: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
@@ -90,18 +89,10 @@ Deploy this bundle from the Juju charm store with the `juju deploy` command:
juju deploy hadoop-kafka
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-kafka`.
-
Alternatively, deploy a locally modified `bundle.yaml` with:
juju deploy /path/to/bundle.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-/path/to/bundle.yaml`.
-
The charms in this bundle can also be built from their source layers in the
[Bigtop charm repository][]. See the [Bigtop charm README][] for instructions
on building and deploying these charms locally.
@@ -113,7 +104,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/hadoop-kafka/bundle.yaml
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -166,25 +156,16 @@ complete. Run the smoke-test actions as follows:
juju run-action kafka/0 smoke-test
juju run-action zookeeper/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do <application>/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, all of the actions should settle to `status: completed`. If
any report `status: failed`, that application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
Applications in this bundle include command line and web utilities that
can be used to verify information about the cluster.
@@ -314,6 +295,18 @@ Multiple units may be added at once. For example, add four more slave units:
juju add-unit -n4 slave
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this bundle at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, bundle issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -324,6 +317,6 @@ Multiple units may be added at once. For example, add four more slave units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-kafka/bundle.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-kafka/bundle.yaml b/bigtop-deploy/juju/hadoop-kafka/bundle.yaml
index 948c889..abf74bb 100644
--- a/bigtop-deploy/juju/hadoop-kafka/bundle.yaml
+++ b/bigtop-deploy/juju/hadoop-kafka/bundle.yaml
@@ -1,6 +1,6 @@
services:
namenode:
- charm: "cs:xenial/hadoop-namenode-13"
+ charm: "cs:xenial/hadoop-namenode-14"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -9,7 +9,7 @@ services:
to:
- "0"
resourcemanager:
- charm: "cs:xenial/hadoop-resourcemanager-14"
+ charm: "cs:xenial/hadoop-resourcemanager-16"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -18,7 +18,7 @@ services:
to:
- "0"
slave:
- charm: "cs:xenial/hadoop-slave-13"
+ charm: "cs:xenial/hadoop-slave-15"
constraints: "mem=7G root-disk=32G"
num_units: 3
annotations:
@@ -29,7 +29,7 @@ services:
- "2"
- "3"
plugin:
- charm: "cs:xenial/hadoop-plugin-13"
+ charm: "cs:xenial/hadoop-plugin-14"
annotations:
gui-x: "1000"
gui-y: "400"
@@ -52,7 +52,7 @@ services:
to:
- "4"
zookeeper:
- charm: "cs:xenial/zookeeper-17"
+ charm: "cs:xenial/zookeeper-19"
constraints: "mem=3G root-disk=32G"
num_units: 3
annotations:
@@ -63,7 +63,7 @@ services:
- "6"
- "7"
kafka:
- charm: "cs:xenial/kafka-12"
+ charm: "cs:xenial/kafka-15"
constraints: "mem=3G"
num_units: 1
annotations:
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-processing/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-processing/README.md b/bigtop-deploy/juju/hadoop-processing/README.md
index 896a793..9ad44b6 100644
--- a/bigtop-deploy/juju/hadoop-processing/README.md
+++ b/bigtop-deploy/juju/hadoop-processing/README.md
@@ -38,10 +38,10 @@ and syslog activity.
The applications that comprise this bundle are spread across 5 machines as
follows:
- * NameNode (HDFS)
- * ResourceManager (YARN)
+ * NameNode v2.7.3
+ * ResourceManager v2.7.3
* Colocated on the NameNode unit
- * Slave (DataNode and NodeManager)
+ * Slave (DataNode and NodeManager) v2.7.3
* 3 separate units
* Client (Hadoop endpoint)
* Plugin (Facilitates communication with the Hadoop cluster)
@@ -57,9 +57,8 @@ demands.
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-bundle.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this bundle.
> **Note**: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
@@ -70,18 +69,10 @@ Deploy this bundle from the Juju charm store with the `juju deploy` command:
juju deploy hadoop-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-processing`.
-
Alternatively, deploy a locally modified `bundle.yaml` with:
juju deploy /path/to/bundle.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-/path/to/bundle.yaml`.
-
The charms in this bundle can also be built from their source layers in the
[Bigtop charm repository][]. See the [Bigtop charm README][] for instructions
on building and deploying these charms locally.
@@ -93,7 +84,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/hadoop-processing/bundle.yaml
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -127,25 +117,16 @@ Run the smoke-test actions as follows:
juju run-action resourcemanager/0 smoke-test
juju run-action slave/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do <application>/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, all of the actions should settle to `status: completed`. If
any report `status: failed`, that application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
Applications in this bundle include Hadoop command line and web utilities that
can be used to verify information about the cluster.
@@ -270,6 +251,18 @@ Multiple units may be added at once. For example, add four more slave units:
juju add-unit -n4 slave
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this bundle at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, bundle issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -280,6 +273,6 @@ Multiple units may be added at once. For example, add four more slave units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-processing/bundle.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-processing/bundle.yaml b/bigtop-deploy/juju/hadoop-processing/bundle.yaml
index c4d8005..ef5195a 100644
--- a/bigtop-deploy/juju/hadoop-processing/bundle.yaml
+++ b/bigtop-deploy/juju/hadoop-processing/bundle.yaml
@@ -1,6 +1,6 @@
services:
namenode:
- charm: "cs:xenial/hadoop-namenode-13"
+ charm: "cs:xenial/hadoop-namenode-14"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -9,7 +9,7 @@ services:
to:
- "0"
resourcemanager:
- charm: "cs:xenial/hadoop-resourcemanager-14"
+ charm: "cs:xenial/hadoop-resourcemanager-16"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -18,7 +18,7 @@ services:
to:
- "0"
slave:
- charm: "cs:xenial/hadoop-slave-13"
+ charm: "cs:xenial/hadoop-slave-15"
constraints: "mem=7G root-disk=32G"
num_units: 3
annotations:
@@ -29,7 +29,7 @@ services:
- "2"
- "3"
plugin:
- charm: "cs:xenial/hadoop-plugin-13"
+ charm: "cs:xenial/hadoop-plugin-14"
annotations:
gui-x: "1000"
gui-y: "400"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-spark/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-spark/README.md b/bigtop-deploy/juju/hadoop-spark/README.md
index cc956e9..13905b3 100644
--- a/bigtop-deploy/juju/hadoop-spark/README.md
+++ b/bigtop-deploy/juju/hadoop-spark/README.md
@@ -26,35 +26,36 @@ to deliver high-availability, Hadoop can detect and handle failures at the
application layer. This provides a highly-available service on top of a cluster
of machines, each of which may be prone to failure.
-Spark is a fast and general engine for large-scale data processing.
+Apache Spark is a fast and general engine for large-scale data processing.
+Learn more at [spark.apache.org][].
This bundle provides a complete deployment of Hadoop and Spark components from
[Apache Bigtop][] that performs distributed data processing at scale. Ganglia
and rsyslog applications are also provided to monitor cluster health and syslog
activity.
+[spark.apache.org]: http://spark.apache.org/
[Apache Bigtop]: http://bigtop.apache.org/
## Bundle Composition
-The applications that comprise this bundle are spread across 9 units as
+The applications that comprise this bundle are spread across 5 units as
follows:
- * NameNode (HDFS)
- * ResourceManager (YARN)
+ * NameNode v2.7.3
+ * ResourceManager v2.7.3
* Colocated on the NameNode unit
- * Slave (DataNode and NodeManager)
- * 3 separate units
- * Spark (Master in yarn-client mode)
- * Zookeeper
+ * Slave (DataNode and NodeManager) v2.7.3
* 3 separate units
+ * Spark (Driver in yarn-client mode) v2.1.0
* Client (Hadoop endpoint)
+ * Colocated on the Spark unit
* Plugin (Facilitates communication with the Hadoop cluster)
- * Colocated on the Spark and Client units
+ * Colocated on the Spark/Client unit
* Ganglia (Web interface for monitoring cluster metrics)
- * Colocated on the Client unit
+ * Colocated on the Spark/Client unit
* Rsyslog (Aggregate cluster syslog events in a single location)
- * Colocated on the Client unit
+ * Colocated on the Spark/Client unit
Deploying this bundle results in a fully configured Apache Bigtop
cluster on any supported cloud, which can be scaled to meet workload
@@ -63,9 +64,8 @@ demands.
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-bundle.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this bundle.
> **Note**: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
@@ -77,18 +77,10 @@ Deploy this bundle from the Juju charm store with the `juju deploy` command:
juju deploy hadoop-spark
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-spark`.
-
Alternatively, deploy a locally modified `bundle.yaml` with:
juju deploy /path/to/bundle.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-/path/to/bundle.yaml`.
-
The charms in this bundle can also be built from their source layers in the
[Bigtop charm repository][]. See the [Bigtop charm README][] for instructions
on building and deploying these charms locally.
@@ -100,7 +92,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/hadoop-spark/bundle.yaml
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -124,8 +115,8 @@ Once they all indicate that they are ready, perform application smoke tests
to verify that the bundle is working as expected.
## Smoke Test
-The charms for each core component (namenode, resourcemanager, slave, spark,
-and zookeeper) provide a `smoke-test` action that can be used to verify the
+The charms for each core component (namenode, resourcemanager, slave, and
+spark) provide a `smoke-test` action that can be used to verify the
application is functioning as expected. Note that the 'slave' component runs
extensive tests provided by Apache Bigtop and may take up to 30 minutes to
complete. Run the smoke-test actions as follows:
@@ -134,27 +125,17 @@ complete. Run the smoke-test actions as follows:
juju run-action resourcemanager/0 smoke-test
juju run-action slave/0 smoke-test
juju run-action spark/0 smoke-test
- juju run-action zookeeper/0 smoke-test
-
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do <application>/0 smoke-test`.
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, all of the actions should settle to `status: completed`. If
any report `status: failed`, that application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
Applications in this bundle include command line and web utilities that
can be used to verify information about the cluster.
@@ -165,10 +146,6 @@ of YARN NodeManager units with the following:
juju run --application namenode "su hdfs -c 'hdfs dfsadmin -report'"
juju run --application resourcemanager "su yarn -c 'yarn node -list'"
-Show the list of Zookeeper nodes with the following:
-
- juju run --unit zookeeper/0 'echo "ls /" | /usr/lib/zookeeper/bin/zkCli.sh'
-
To access the HDFS web console, find the `PUBLIC-ADDRESS` of the namenode
application and expose it:
@@ -204,7 +181,7 @@ The web interface will be available at the following URL:
# Monitoring
This bundle includes Ganglia for system-level monitoring of the namenode,
-resourcemanager, slave, spark, and zookeeper units. Metrics are sent to a
+resourcemanager, slave, and spark units. Metrics are sent to a
centralized ganglia unit for easy viewing in a browser. To view the ganglia web
interface, find the `PUBLIC-ADDRESS` of the Ganglia application and expose it:
@@ -219,7 +196,7 @@ The web interface will be available at:
# Logging
This bundle includes rsyslog to collect syslog data from the namenode,
-resourcemanager, slave, spark, and zookeeper units. These logs are sent to a
+resourcemanager, slave, and spark units. These logs are sent to a
centralized rsyslog unit for easy syslog analysis. One method of viewing this
log data is to simply cat syslog from the rsyslog unit:
@@ -278,27 +255,17 @@ run with `juju run-action`:
enqueued: 2016-02-04 14:55:14 +0000 UTC
started: 2016-02-04 14:55:27 +0000 UTC
-The `spark` charm in this bundle also provides several benchmarks to gauge
-the performance of the Spark cluster. Each benchmark is an action that can be
-run with `juju run-action`:
+The `spark` charm in this bundle provides benchmarks to gauge the performance
+of the Spark/YARN cluster. Each benchmark is an action that can be run with
+`juju run-action`:
- $ juju actions spark | grep Bench
- connectedcomponent Run the Spark Bench ConnectedComponent benchmark.
- decisiontree Run the Spark Bench DecisionTree benchmark.
- kmeans Run the Spark Bench KMeans benchmark.
- linearregression Run the Spark Bench LinearRegression benchmark.
- logisticregression Run the Spark Bench LogisticRegression benchmark.
- matrixfactorization Run the Spark Bench MatrixFactorization benchmark.
- pagerank Run the Spark Bench PageRank benchmark.
- pca Run the Spark Bench PCA benchmark.
- pregeloperation Run the Spark Bench PregelOperation benchmark.
- shortestpaths Run the Spark Bench ShortestPaths benchmark.
- sql Run the Spark Bench SQL benchmark.
- stronglyconnectedcomponent Run the Spark Bench StronglyConnectedComponent benchmark.
- svdplusplus Run the Spark Bench SVDPlusPlus benchmark.
- svm Run the Spark Bench SVM benchmark.
-
- $ juju run-action spark/0 svdplusplus
+ $ juju actions spark
+ ...
+ pagerank Calculate PageRank for a sample data set
+ sparkpi Calculate Pi
+ ...
+
+ $ juju run-action spark/0 pagerank
Action queued with id: 339cec1f-e903-4ee7-85ca-876fb0c3d28e
$ juju show-action-output 339cec1f-e903-4ee7-85ca-876fb0c3d28e
@@ -307,40 +274,41 @@ run with `juju run-action`:
composite:
direction: asc
units: secs
- value: "200.754000"
- raw: |
- SVDPlusPlus,2016-11-02-03:08:26,200.754000,85.974071,.428255,0,SVDPlusPlus-MLlibConfig,,,,,10,,,50000,4.0,1.3,
- start: 2016-11-02T03:08:26Z
- stop: 2016-11-02T03:11:47Z
- results:
- duration:
- direction: asc
- units: secs
- value: "200.754000"
- throughput:
- direction: desc
- units: MB/sec
- value: ".428255"
+ value: "83"
+ start: 2017-04-12T23:22:38Z
+ stop: 2017-04-12T23:24:01Z
+ output: '{''status'': ''completed''}'
status: completed
timing:
- completed: 2016-11-02 03:11:48 +0000 UTC
- enqueued: 2016-11-02 03:08:21 +0000 UTC
- started: 2016-11-02 03:08:26 +0000 UTC
+ completed: 2017-04-12 23:24:02 +0000 UTC
+ enqueued: 2017-04-12 23:22:36 +0000 UTC
+ started: 2017-04-12 23:22:37 +0000 UTC
# Scaling
-By default, three Hadoop slave and three zookeeper units are deployed. Scaling
-these applications is as simple as adding more units. To add one unit:
+By default, three Hadoop slave units are deployed. Scaling these is as simple
+as adding more units. To add one unit:
juju add-unit slave
- juju add-unit zookeeper
Multiple units may be added at once. For example, add four more slave units:
juju add-unit -n4 slave
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this bundle at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, bundle issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -351,6 +319,6 @@ Multiple units may be added at once. For example, add four more slave units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-spark/bundle.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-spark/bundle.yaml b/bigtop-deploy/juju/hadoop-spark/bundle.yaml
index 500ae78..0611495 100644
--- a/bigtop-deploy/juju/hadoop-spark/bundle.yaml
+++ b/bigtop-deploy/juju/hadoop-spark/bundle.yaml
@@ -1,6 +1,6 @@
services:
namenode:
- charm: "cs:xenial/hadoop-namenode-13"
+ charm: "cs:xenial/hadoop-namenode-14"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -9,7 +9,7 @@ services:
to:
- "0"
resourcemanager:
- charm: "cs:xenial/hadoop-resourcemanager-14"
+ charm: "cs:xenial/hadoop-resourcemanager-16"
constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
@@ -18,7 +18,7 @@ services:
to:
- "0"
slave:
- charm: "cs:xenial/hadoop-slave-13"
+ charm: "cs:xenial/hadoop-slave-15"
constraints: "mem=7G root-disk=32G"
num_units: 3
annotations:
@@ -29,13 +29,13 @@ services:
- "2"
- "3"
plugin:
- charm: "cs:xenial/hadoop-plugin-13"
+ charm: "cs:xenial/hadoop-plugin-14"
annotations:
gui-x: "1000"
gui-y: "400"
client:
charm: "cs:xenial/hadoop-client-3"
- constraints: "mem=3G"
+ constraints: "mem=7G root-disk=32G"
num_units: 1
annotations:
gui-x: "1250"
@@ -43,7 +43,7 @@ services:
to:
- "4"
spark:
- charm: "cs:xenial/spark-31"
+ charm: "cs:xenial/spark-34"
constraints: "mem=7G root-disk=32G"
num_units: 1
options:
@@ -52,18 +52,7 @@ services:
gui-x: "1000"
gui-y: "0"
to:
- - "5"
- zookeeper:
- charm: "cs:xenial/zookeeper-17"
- constraints: "mem=3G root-disk=32G"
- num_units: 3
- annotations:
- gui-x: "500"
- gui-y: "400"
- to:
- - "6"
- - "7"
- - "8"
+ - "4"
ganglia:
charm: "cs:xenial/ganglia-12"
num_units: 1
@@ -97,20 +86,17 @@ relations:
- [resourcemanager, slave]
- [plugin, namenode]
- [plugin, resourcemanager]
- - [client, plugin]
- [spark, plugin]
- - [spark, zookeeper]
+ - [client, plugin]
- ["ganglia-node:juju-info", "namenode:juju-info"]
- ["ganglia-node:juju-info", "resourcemanager:juju-info"]
- ["ganglia-node:juju-info", "slave:juju-info"]
- ["ganglia-node:juju-info", "spark:juju-info"]
- - ["ganglia-node:juju-info", "zookeeper:juju-info"]
- ["ganglia:node", "ganglia-node:node"]
- ["rsyslog-forwarder-ha:juju-info", "namenode:juju-info"]
- ["rsyslog-forwarder-ha:juju-info", "resourcemanager:juju-info"]
- ["rsyslog-forwarder-ha:juju-info", "slave:juju-info"]
- ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"]
- - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"]
- ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"]
machines:
"0":
@@ -127,16 +113,4 @@ machines:
constraints: "mem=7G root-disk=32G"
"4":
series: "xenial"
- constraints: "mem=3G"
- "5":
- series: "xenial"
constraints: "mem=7G root-disk=32G"
- "6":
- series: "xenial"
- constraints: "mem=3G root-disk=32G"
- "7":
- series: "xenial"
- constraints: "mem=3G root-disk=32G"
- "8":
- series: "xenial"
- constraints: "mem=3G root-disk=32G"
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/hadoop-spark/tests/tests.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/hadoop-spark/tests/tests.yaml b/bigtop-deploy/juju/hadoop-spark/tests/tests.yaml
index b951742..cb745df 100644
--- a/bigtop-deploy/juju/hadoop-spark/tests/tests.yaml
+++ b/bigtop-deploy/juju/hadoop-spark/tests/tests.yaml
@@ -5,9 +5,15 @@ sources:
packages:
- amulet
- python3-yaml
-# exclude tests that are unrelated to bigtop.
+# exclude tests that are unrelated to bigtop. the exclusion of spark might
+# look weird here, but for this bundle, we only care that spark is good in
+# yarn mode (covered by this bundle when we invoke the spark smoke-test). the
+# typical spark tests will test spark once in standalone and twice more in
+# various HA modes. that takes forever, so leave those heavy tests for the
+# spark-processing bundle. let's go fast on this one.
excludes:
- ganglia
- ganglia-node
- rsyslog
- rsyslog-forwarder-ha
+ - spark
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/spark-processing/README.md
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/spark-processing/README.md b/bigtop-deploy/juju/spark-processing/README.md
index c39fa2c..a499a38 100644
--- a/bigtop-deploy/juju/spark-processing/README.md
+++ b/bigtop-deploy/juju/spark-processing/README.md
@@ -16,12 +16,14 @@
-->
# Overview
-This bundle provides a complete deployment of
-[Apache Spark][] in standalone HA mode as provided
-by [Apache Bigtop][]. Ganglia and rsyslog
+Apache Spark is a fast and general engine for large-scale data processing.
+Learn more at [spark.apache.org][].
+
+This bundle provides a complete deployment of Spark (in standalone HA mode)
+and Apache Zookeeper components from [Apache Bigtop][]. Ganglia and rsyslog
applications are included to monitor cluster health and syslog activity.
-[Apache Spark]: http://spark/apache.org/
+[spark.apache.org]: http://spark.apache.org/
[Apache Bigtop]: http://bigtop.apache.org/
## Bundle Composition
@@ -29,9 +31,9 @@ applications are included to monitor cluster health and syslog activity.
The applications that comprise this bundle are spread across 6 units as
follows:
- * Spark (Master and Worker)
+ * Spark (Master and Worker) v2.1.0
* 2 separate units
- * Zookeeper
+ * Zookeeper v3.4.6
* 3 separate units
* Ganglia (Web interface for monitoring cluster metrics)
* Rsyslog (Aggregate cluster syslog events in a single location)
@@ -44,9 +46,8 @@ demands.
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-bundle.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this bundle.
> **Note**: This bundle requires hardware resources that may exceed limits
of Free-tier or Trial accounts on some clouds. To deploy to these
@@ -58,18 +59,10 @@ Deploy this bundle from the Juju charm store with the `juju deploy` command:
juju deploy spark-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-spark-processing`.
-
Alternatively, deploy a locally modified `bundle.yaml` with:
juju deploy /path/to/bundle.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-/path/to/bundle.yaml`.
-
The charms in this bundle can also be built from their source layers in the
[Bigtop charm repository][]. See the [Bigtop charm README][] for instructions
on building and deploying these charms locally.
@@ -81,7 +74,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/spark-processing/bundle.yaml
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
[Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -112,25 +104,16 @@ actions as follows:
juju run-action spark/0 smoke-test
juju run-action zookeeper/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do <application>/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, all of the actions should settle to `status: completed`. If
any report `status: failed`, that application is not working as expected. Get
more information about the smoke-test action
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
Applications in this bundle include Zookeeper command line and Spark web
utilities that can be used to verify information about the cluster.
@@ -181,27 +164,17 @@ the [rsyslog README](https://jujucharms.com/rsyslog/) for more information.
# Benchmarking
-The `spark` charm in this bundle provides several benchmarks to gauge
-the performance of the Spark cluster. Each benchmark is an action that can be
-run with `juju run-action`:
-
- $ juju actions spark | grep Bench
- connectedcomponent Run the Spark Bench ConnectedComponent benchmark.
- decisiontree Run the Spark Bench DecisionTree benchmark.
- kmeans Run the Spark Bench KMeans benchmark.
- linearregression Run the Spark Bench LinearRegression benchmark.
- logisticregression Run the Spark Bench LogisticRegression benchmark.
- matrixfactorization Run the Spark Bench MatrixFactorization benchmark.
- pagerank Run the Spark Bench PageRank benchmark.
- pca Run the Spark Bench PCA benchmark.
- pregeloperation Run the Spark Bench PregelOperation benchmark.
- shortestpaths Run the Spark Bench ShortestPaths benchmark.
- sql Run the Spark Bench SQL benchmark.
- stronglyconnectedcomponent Run the Spark Bench StronglyConnectedComponent benchmark.
- svdplusplus Run the Spark Bench SVDPlusPlus benchmark.
- svm Run the Spark Bench SVM benchmark.
-
- $ juju run-action spark/0 svdplusplus
+The `spark` charm in this bundle provides benchmarks to gauge the performance
+of the Spark cluster. Each benchmark is an action that can be run with
+`juju run-action`:
+
+ $ juju actions spark
+ ...
+ pagerank Calculate PageRank for a sample data set
+ sparkpi Calculate Pi
+ ...
+
+ $ juju run-action spark/0 pagerank
Action queued with id: 339cec1f-e903-4ee7-85ca-876fb0c3d28e
$ juju show-action-output 339cec1f-e903-4ee7-85ca-876fb0c3d28e
@@ -210,25 +183,15 @@ run with `juju run-action`:
composite:
direction: asc
units: secs
- value: "200.754000"
- raw: |
- SVDPlusPlus,2016-11-02-03:08:26,200.754000,85.974071,.428255,0,SVDPlusPlus-MLlibConfig,,,,,10,,,50000,4.0,1.3,
- start: 2016-11-02T03:08:26Z
- stop: 2016-11-02T03:11:47Z
- results:
- duration:
- direction: asc
- units: secs
- value: "200.754000"
- throughput:
- direction: desc
- units: x/sec
- value: ".428255"
+ value: "83"
+ start: 2017-04-12T23:22:38Z
+ stop: 2017-04-12T23:24:01Z
+ output: '{''status'': ''completed''}'
status: completed
timing:
- completed: 2016-11-02 03:11:48 +0000 UTC
- enqueued: 2016-11-02 03:08:21 +0000 UTC
- started: 2016-11-02 03:08:26 +0000 UTC
+ completed: 2017-04-12 23:24:02 +0000 UTC
+ enqueued: 2017-04-12 23:22:36 +0000 UTC
+ started: 2017-04-12 23:22:37 +0000 UTC
# Scaling
@@ -244,6 +207,18 @@ Multiple units may be added at once. For example, add four more spark units:
juju add-unit -n4 spark
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this bundle at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, bundle issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -254,6 +229,6 @@ Multiple units may be added at once. For example, add four more spark units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/spark-processing/bundle.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/spark-processing/bundle.yaml b/bigtop-deploy/juju/spark-processing/bundle.yaml
index bb79d94..f114ebf 100644
--- a/bigtop-deploy/juju/spark-processing/bundle.yaml
+++ b/bigtop-deploy/juju/spark-processing/bundle.yaml
@@ -1,6 +1,6 @@
services:
spark:
- charm: "cs:xenial/spark-31"
+ charm: "cs:xenial/spark-34"
constraints: "mem=7G root-disk=32G"
num_units: 2
options:
@@ -13,7 +13,7 @@ services:
- "0"
- "1"
zookeeper:
- charm: "cs:xenial/zookeeper-17"
+ charm: "cs:xenial/zookeeper-19"
constraints: "mem=3G root-disk=32G"
num_units: 3
annotations:
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-deploy/juju/spark-processing/tests/tests.yaml
----------------------------------------------------------------------
diff --git a/bigtop-deploy/juju/spark-processing/tests/tests.yaml b/bigtop-deploy/juju/spark-processing/tests/tests.yaml
index e4b472e..b951742 100644
--- a/bigtop-deploy/juju/spark-processing/tests/tests.yaml
+++ b/bigtop-deploy/juju/spark-processing/tests/tests.yaml
@@ -5,15 +5,9 @@ sources:
packages:
- amulet
- python3-yaml
-# exclude tests that are unrelated to bigtop. the exclusion of spark might
-# look weird here, but for this bundle, we only care that spark is good in
-# HA mode (covered by this bundle when we invoke the spark smoke-test). the
-# typical spark tests will test spark once in standalone and twice more in
-# various HA modes. that takes forever, so leave those heavy tests for the
-# hadoop-spark bundle. let's go fast on this one.
+# exclude tests that are unrelated to bigtop.
excludes:
- ganglia
- ganglia-node
- rsyslog
- rsyslog-forwarder-ha
- - spark
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/README.md
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/README.md b/bigtop-packages/src/charm/README.md
index 04213ad..cdc50e4 100644
--- a/bigtop-packages/src/charm/README.md
+++ b/bigtop-packages/src/charm/README.md
@@ -28,9 +28,10 @@ Bigtop components. Built charms are published to the [Juju charm store][]
and can be deployed directly from there, either individually or with
[bundles][]. They can also be built from these layers and deployed locally.
-For the remainder of this guide, a working Juju installation is assumed to be
-present. If Juju is not yet set up, please follow the [getting-started][]
-instructions prior to deploying locally built charms and bundles.
+For the remainder of this guide, a working Juju 2.0 or greater installation is
+assumed to be present. If Juju is not yet set up, please follow the
+[getting-started][] instructions prior to deploying locally built charms and
+bundles.
[Juju charm store]: https://jujucharms.com/
[bundles]: https://jujucharms.com/hadoop-processing
@@ -48,11 +49,12 @@ For example:
mkdir $JUJU_REPOSITORY
cd bigtop-packages/src/charms/hadoop/layer-hadoop-namenode
- charm build
+ charm build --series xenial --report
-This will build the NameNode charm, pulling in the appropriate base and
-interface layers from [interfaces.juju.solutions][]. You can get local copies
-of those layers as well by using `charm pull-source`:
+This will build the NameNode charm in the
+$JUJU_REPOSITORY/xenial/hadoop-namenode directory, pulling in the appropriate
+base and interface layers from [interfaces.juju.solutions][]. You can get
+local copies of those layers as well by using `charm pull-source`:
export LAYER_PATH=$HOME/layers
export INTERFACE_PATH=$HOME/interfaces
@@ -65,17 +67,10 @@ You can deploy the locally built charms individually, for example:
juju deploy $JUJU_REPOSITORY/xenial/hadoop-namenode
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is: `juju deploy local:xenial/hadoop-namenode`.
-
You can also deploy the local version of a bundle:
juju deploy ./bigtop-deploy/juju/hadoop-processing/bundle-local.yaml
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-./bigtop-deploy/juju/hadoop-processing/bundle-local.yaml`.
-
[charm-tools]: https://jujucharms.com/docs/stable/tools-charm-tools
[Getting Started]: https://jujucharms.com/docs/stable/developer-getting-started
[interfaces.juju.solutions]: http://interfaces.juju.solutions/
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/README.md
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/README.md b/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/README.md
index e5cbf63..b92ab41 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/README.md
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/README.md
@@ -20,27 +20,22 @@ The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.
-This charm deploys the NameNode component of the [Apache Bigtop][] platform
-to provide HDFS master resources.
+This charm deploys version 2.7.3 of the NameNode component from
+[Apache Bigtop][].
[Apache Bigtop]: http://bigtop.apache.org/
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-charm.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this charm.
This charm is intended to be deployed via one of the [apache bigtop bundles][].
For example:
juju deploy hadoop-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-processing`.
-
This will deploy an Apache Bigtop cluster with this charm acting as the
NameNode. More information about this deployment can be found in the
[bundle readme](https://jujucharms.com/hadoop-processing/).
@@ -52,7 +47,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -79,25 +73,16 @@ application is functioning as expected. Run the action as follows:
juju run-action namenode/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do namenode/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, the action should settle to `status: completed`. If it
reports `status: failed`, the application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
This charm includes Hadoop command line and web utilities that can be used
to verify information about the cluster.
@@ -117,6 +102,18 @@ The web interface will be available at the following URL:
http://NAMENODE_PUBLIC_IP:50070
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this charm at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, charm issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -127,6 +124,6 @@ The web interface will be available at the following URL:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/metadata.yaml
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/metadata.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/metadata.yaml
index a358a6d..2a1806f 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/metadata.yaml
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-namenode/metadata.yaml
@@ -1,11 +1,12 @@
name: hadoop-namenode
-summary: HDFS master (NameNode) from Apache Bigtop
+summary: HDFS NameNode from Apache Bigtop
maintainer: Juju Big Data <bi...@lists.ubuntu.com>
description: >
Hadoop is a software platform that lets one easily write and
run applications that process vast amounts of data.
- This charm provides the HDFS master node (NameNode).
+ This charm provides version 2.7.3 of the HDFS NameNode application from
+ Apache Bigtop.
tags: []
provides:
namenode:
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md
index f9e4483..8c9c032 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md
@@ -28,19 +28,14 @@ This charm facilitates communication between Hadoop components of an
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-charm.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this charm.
This charm is intended to be deployed via one of the [apache bigtop bundles][].
For example:
juju deploy hadoop-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-processing`.
-
This will deploy an Apache Bigtop cluster with a client unit preconfigured to
work with the cluster. More information about this deployment can be found in the
[bundle readme](https://jujucharms.com/hadoop-processing/).
@@ -52,7 +47,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -79,25 +73,16 @@ application is functioning as expected. Run the action as follows:
juju run-action plugin/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do plugin/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, the action should settle to `status: completed`. If it
reports `status: failed`, the application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
This charm includes Hadoop command line utilities that can be used
to verify information about the cluster.
@@ -107,6 +92,18 @@ Show the dfsadmin report on the command line with the following:
juju run --application plugin "su hdfs -c 'hdfs dfsadmin -report'"
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this charm at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, charm issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -117,6 +114,6 @@ Show the dfsadmin report on the command line with the following:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/README.md
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/README.md b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/README.md
index 829ea3a..bd82d00 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/README.md
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/README.md
@@ -20,27 +20,22 @@ The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.
-This charm deploys the ResourceManager component of the [Apache Bigtop][]
-platform to provide YARN master resources.
+This charm deploys version 2.7.3 of the ResourceManager component from
+[Apache Bigtop][].
[Apache Bigtop]: http://bigtop.apache.org/
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-charm.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this charm.
This charm is intended to be deployed via one of the [apache bigtop bundles][].
For example:
juju deploy hadoop-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-processing`.
-
This will deploy an Apache Bigtop cluster with this charm acting as the
ResourceManager. More information about this deployment can be found in the
[bundle readme](https://jujucharms.com/hadoop-processing/).
@@ -52,7 +47,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -81,25 +75,16 @@ smoke tests provided by Apache Bigtop and may take up to
juju run-action resourcemanager/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do resourcemanager/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, the action should settle to `status: completed`. If it
reports `status: failed`, the application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
## Utilities
This charm includes Hadoop command line and web utilities that can be used
to verify information about the cluster.
@@ -168,6 +153,18 @@ cluster. Each benchmark is an action that can be run with `juju run-action`:
started: 2016-02-04 14:55:27 +0000 UTC
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this charm at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, charm issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -178,6 +175,6 @@ cluster. Each benchmark is an action that can be run with `juju run-action`:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/mrbench
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/mrbench b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/mrbench
index d177a2d..2b97413 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/mrbench
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/mrbench
@@ -71,6 +71,7 @@ benchmark-finish
cat ${RESULT_LOG} | $CHARM_DIR/actions/parseBenchmark.py
DURATION=`expr $STOP - $START`
benchmark-composite "${DURATION}" 'secs' 'asc'
+action-set outcome="success"
# clean out benchmark dir
benchmark_cleanup
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/nnbench
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/nnbench b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/nnbench
index 13d7618..64bb230 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/nnbench
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/nnbench
@@ -75,6 +75,7 @@ benchmark-finish
cat ${RESULT_LOG} | $CHARM_DIR/actions/parseBenchmark.py
DURATION=`expr $STOP - $START`
benchmark-composite "${DURATION}" 'secs' 'asc'
+action-set outcome="success"
# clean out benchmark dir
benchmark_cleanup
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/parseBenchmark.py
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/parseBenchmark.py b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/parseBenchmark.py
index 4abdea0..2232aa5 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/parseBenchmark.py
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/parseBenchmark.py
@@ -37,7 +37,8 @@ def parse_benchmark_output():
m = regex.match(line)
if m:
results[m.group(1)] = m.group(2)
- hookenv.action_set({"results.raw": json.dumps(results)})
+ hookenv.action_set({"meta.raw": json.dumps(results)})
+
if __name__ == "__main__":
parse_benchmark_output()
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/smoke-test
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/smoke-test b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/smoke-test
index 4218f7e..34f160b 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/smoke-test
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/smoke-test
@@ -23,12 +23,12 @@ from charms.layer.apache_bigtop_base import Bigtop # noqa: E402
from charms.reactive import is_state # noqa: E402
-def fail(msg, output=None):
- if output:
- hookenv.action_set({'output': output})
+def fail(msg):
+ hookenv.action_set({'outcome': 'failure'})
hookenv.action_fail(msg)
sys.exit()
+
if not is_state('apache-bigtop-resourcemanager.ready'):
fail('Charm is not yet ready to run the Bigtop smoke test(s)')
@@ -43,6 +43,6 @@ smoke_env = {
bigtop = Bigtop()
result = bigtop.run_smoke_tests(smoke_components, smoke_env)
if result == 'success':
- hookenv.action_set({'outcome': result})
+ hookenv.action_set({'outcome': 'success'})
else:
- fail('{} smoke tests failed'.format(smoke_components), result)
+ fail('{} smoke tests failed with: {}'.format(smoke_components, result))
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/teragen
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/teragen b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/teragen
index d08468c..95a65e2 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/teragen
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/teragen
@@ -57,6 +57,7 @@ benchmark-finish
cat ${RESULT_LOG} | $CHARM_DIR/actions/parseBenchmark.py
DURATION=`expr $STOP - $START`
benchmark-composite "${DURATION}" 'secs' 'asc'
+action-set outcome="success"
# clean out benchmark dir
benchmark_cleanup
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/terasort
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/terasort b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/terasort
index 5e3e592..2611318 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/terasort
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/terasort
@@ -78,6 +78,7 @@ benchmark-finish
cat ${RESULT_LOG} | $CHARM_DIR/actions/parseBenchmark.py
DURATION=`expr $STOP - $START`
benchmark-composite "${DURATION}" 'secs' 'asc'
+action-set outcome="success"
# clean out benchmark dir
benchmark_cleanup
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/testdfsio
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/testdfsio b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/testdfsio
index 558c9f4..a53ccac 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/testdfsio
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/actions/testdfsio
@@ -76,6 +76,7 @@ benchmark-finish
cat ${RESULT_LOG} | $CHARM_DIR/actions/parseBenchmark.py
DURATION=`expr $STOP - $START`
benchmark-composite "${DURATION}" 'secs' 'asc'
+action-set outcome="success"
# clean out benchmark dir
benchmark_cleanup
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/metadata.yaml
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/metadata.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/metadata.yaml
index 695d5bf..84e4255 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/metadata.yaml
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-resourcemanager/metadata.yaml
@@ -1,11 +1,12 @@
name: hadoop-resourcemanager
-summary: YARN master (ResourceManager) from Apache Bigtop
+summary: YARN ResourceManager from Apache Bigtop
maintainer: Juju Big Data <bi...@lists.ubuntu.com>
description: >
Hadoop is a software platform that lets one easily write and
run applications that process vast amounts of data.
- This charm provides the YARN master node (ResourceManager).
+ This charm provides version 2.7.3 of the YARN ResourceManager application from
+ Apache Bigtop.
tags: []
provides:
resourcemanager:
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/README.md
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/README.md b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/README.md
index ee93239..41635dc 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/README.md
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/README.md
@@ -20,28 +20,22 @@ The Apache Hadoop software library is a framework that allows for the
distributed processing of large data sets across clusters of computers
using a simple programming model.
-This charm deploys a combined slave node running the NodeManager
-and DataNode components of the [Apache Bigtop][] platform
-to provide YARN compute and HDFS storage resources.
+This charm deploys a combined slave node running version 2.7.3 of both the
+NodeManager and DataNode components from [Apache Bigtop][].
[Apache Bigtop]: http://bigtop.apache.org/
# Deploying
-A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the [getting-started][] instructions prior to deploying this
-charm.
+This charm requires Juju 2.0 or greater. If Juju is not yet set up, please
+follow the [getting-started][] instructions prior to deploying this charm.
This charm is intended to be deployed via one of the [apache bigtop bundles][].
For example:
juju deploy hadoop-processing
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
-hadoop-processing`.
-
This will deploy an Apache Bigtop cluster with 3 units of this charm acting as
the combined DataNode/NodeManager application. More information about this
deployment can be found in the [bundle readme](https://jujucharms.com/hadoop-processing/).
@@ -53,7 +47,6 @@ mirror options. See [Configuring Models][] for more information.
[getting-started]: https://jujucharms.com/docs/stable/getting-started
[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles
-[juju-quickstart]: https://launchpad.net/juju-quickstart
[Configuring Models]: https://jujucharms.com/docs/stable/models-config
@@ -82,25 +75,16 @@ and 'mapreduce' smoke tests provided by Apache Bigtop and may take up to
juju run-action slave/0 smoke-test
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do slave/0 smoke-test`.
-
Watch the progress of the smoke test actions with:
watch -n 2 juju show-action-status
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action status`.
-
Eventually, the action should settle to `status: completed`. If it
reports `status: failed`, the application is not working as expected. Get
more information about a specific smoke test with:
juju show-action-output <action-id>
-> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action fetch <action-id>`.
-
# Scaling
@@ -114,6 +98,18 @@ Multiple units may be added at once. For example, add four more slave units:
juju add-unit -n4 slave
+# Issues
+
+Apache Bigtop tracks issues using JIRA (Apache account required). File an
+issue for this charm at:
+
+https://issues.apache.org/jira/secure/CreateIssue!default.jspa
+
+Ensure `Bigtop` is selected as the project. Typically, charm issues are filed
+in the `deployment` component with the latest stable release selected as the
+affected version. Any uncertain fields may be left blank.
+
+
# Contact Information
- <bi...@lists.ubuntu.com>
@@ -124,6 +120,6 @@ Multiple units may be added at once. For example, add four more slave units:
- [Apache Bigtop home page](http://bigtop.apache.org/)
- [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html)
- [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html)
-- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop)
+- [Juju Big Data](https://jujucharms.com/big-data)
+- [Juju Bigtop charms](https://jujucharms.com/q/bigtop)
- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju)
-- [Juju community](https://jujucharms.com/community)
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/actions/smoke-test
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/actions/smoke-test b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/actions/smoke-test
index 68e646d..f1bb34d 100755
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/actions/smoke-test
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/actions/smoke-test
@@ -23,12 +23,12 @@ from charms.layer.apache_bigtop_base import Bigtop # noqa: E402
from charms.reactive import is_state # noqa: E402
-def fail(msg, output=None):
- if output:
- hookenv.action_set({'output': output})
+def fail(msg):
+ hookenv.action_set({'outcome': 'failure'})
hookenv.action_fail(msg)
sys.exit()
+
if not is_state('apache-bigtop-datanode.started'):
fail('Charm is not yet ready to run the Bigtop smoke test(s)')
@@ -44,6 +44,6 @@ smoke_env = {
bigtop = Bigtop()
result = bigtop.run_smoke_tests(smoke_components, smoke_env)
if result == 'success':
- hookenv.action_set({'outcome': result})
+ hookenv.action_set({'outcome': 'success'})
else:
- fail('{} smoke tests failed'.format(smoke_components), result)
+ fail('{} smoke tests failed with: {}'.format(smoke_components, result))
http://git-wip-us.apache.org/repos/asf/bigtop/blob/b5b366fe/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/metadata.yaml
----------------------------------------------------------------------
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/metadata.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/metadata.yaml
index 9e4601e..0b56f32 100644
--- a/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/metadata.yaml
+++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-slave/metadata.yaml
@@ -5,5 +5,5 @@ description: >
Hadoop is a software platform that lets one easily write and
run applications that process vast amounts of data.
- This charm provides both the storage node (DataNode) for HDFS and the
- compute node (NodeManager) for Yarn.
+ This charm provides version 2.7.3 of both the HDFS DataNode and the
+ YARN NodeManager from Apache Bigtop.