You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@heron.apache.org by GitBox <gi...@apache.org> on 2018/05/17 14:45:58 UTC
[GitHub] kramasamy closed pull request #2898: add documentation about nomad
consul integration
kramasamy closed pull request #2898: add documentation about nomad consul integration
URL: https://github.com/apache/incubator-heron/pull/2898
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/heron/config/src/yaml/conf/nomad/scheduler.yaml b/heron/config/src/yaml/conf/nomad/scheduler.yaml
index 8e7e3a5cf8..832ca76164 100644
--- a/heron/config/src/yaml/conf/nomad/scheduler.yaml
+++ b/heron/config/src/yaml/conf/nomad/scheduler.yaml
@@ -58,5 +58,5 @@ heron.nomad.metrics.service.check.interval.sec: 10
heron.nomad.metrics.service.check.timeout.sec: 2
# additional tags to be attached to metrics service
-# A tag of <topology-name>-<container-index> with be automaticallu attached
-heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
\ No newline at end of file
+# A tag of <topology-name>-<container-index> with be automatically attached
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
diff --git a/heron/config/src/yaml/conf/standalone/scheduler.yaml b/heron/config/src/yaml/conf/standalone/scheduler.yaml
index fc1cd27a94..a492ef2e86 100644
--- a/heron/config/src/yaml/conf/standalone/scheduler.yaml
+++ b/heron/config/src/yaml/conf/standalone/scheduler.yaml
@@ -51,5 +51,5 @@ heron.nomad.metrics.service.check.interval.sec: 10
heron.nomad.metrics.service.check.timeout.sec: 2
# additional tags to be attached to metrics service in a comma delimited list
-# A tag of <topology-name>-<container-index> with be automaticallu attached
+# A tag of <topology-name>-<container-index> with be automatically attached
heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
diff --git a/website/content/docs/operators/deployment/schedulers/nomad-docker.md b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
index 063a4cc993..cde8320573 100644
--- a/website/content/docs/operators/deployment/schedulers/nomad-docker.md
+++ b/website/content/docs/operators/deployment/schedulers/nomad-docker.md
@@ -4,7 +4,7 @@ title: Running Heron via Docker Containers on Nomad
Below are instructions on how to run Heron on Nomad via docker containers. In this mode, Heron executors will run as docker containers on host machines.
-### Requirements
+## Requirements
When setting up your Nomad cluster, the following are required:
@@ -14,7 +14,7 @@ When setting up your Nomad cluster, the following are required:
* Docker installed and enabled on every machine
* Each machine must also be able to pull the official Heron docker image from DockerHub or have the image preloaded.
-### Configuring Heron settings
+## Configuring Heron settings
Before running Heron via Nomad, you'll need to configure some settings. Once you've [installed Heron](../../../../getting-started), all of the configurations you'll need to modify will be in the `~/.heron/conf/nomad` diredctory.
@@ -132,3 +132,33 @@ Heron users can upload their Heron topology packages to the Heron API server usi
heron.class.uploader: org.apache.heron.uploader.http.HttpUploader
heron.uploader.http.uri: http://localhost:9000/api/v1/file/upload
```
+
+## Integration with Consul for metrics
+Each container part of a Heron topology serves metrics out of a port randomly generated by Nomad. Thus, Consul is needed for service discovery for users to determine which port the container is serving the metrics out of.
+Every Heron executor running in a docker container will automatically register itself as a service with Consul given that there is a Consul cluster running. The port Heron will be serving metrics will be registered with Consul.
+
+The service will be registered with the name with the following format:
+
+```yaml
+metrics-heron-<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+Each heron executor registered with Consul will be tagged with
+
+```yaml
+<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+To add additional tags, please add specify them in a comma delimited list via
+
+```yaml
+heron.nomad.metrics.service.additional.tags
+```
+
+in `scheduler.yaml`. For example:
+
+```yaml
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
+```
+
+Users can then configure Prometheus to scrape metrics for each container based on these tags
diff --git a/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md b/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
index b362123874..79cc02afcc 100644
--- a/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
+++ b/website/content/docs/operators/deployment/schedulers/nomad-raw-execs.md
@@ -6,7 +6,7 @@ Below are instructions on how to to run Heron on Nomad via raw execs. In this m
The advantages of this mode is that it is incredibly lightweight and likely do not require sudo privileges to setup and run. However in this mode, the setup procedure may be a little more complex compared to running via docker since there are more things to consider. Also in resource allocation is considered but not enforced.
-### Requirements
+## Requirements
When setting up your Nomad cluster, the following are required:
@@ -14,7 +14,7 @@ When setting up your Nomad cluster, the following are required:
* Python 2.7, Java 7 or 8, and [curl](https://curl.haxx.se/) must be installed on every machine in the cluster
* A [ZooKeeper cluster](https://zookeeper.apache.org)
-### Configuring Heron settings
+## Configuring Heron settings
Before running Heron via Nomad, you'll need to configure some settings. Once you've [installed Heron](../../../../getting-started), all of the configurations you'll need to modify will be in the `~/.heron/conf/nomad` diredctory.
@@ -52,7 +52,7 @@ heron.scheduler.local.working.directory: ${HOME}/.herondata/topologies/${CLUSTER
> Heron uses string interpolation to fill in the missing values for `CLUSTER`, `ROLE`, etc.
-### Distributing Heron core
+## Distributing Heron core
The Heron core package needs to be made available for every machine in the cluster to download. You'll need to provide a URI for the Heron core package. Here are the currently supported protocols:
@@ -214,3 +214,33 @@ $ heron submit nomad \
org.apache.heron.examples.api.WindowedWordCountTopology \
windowed-word-count
```
+
+## Integration with Consul for metrics
+Each Heron executor part of a Heron topology serves metrics out of a port randomly generated by Nomad. Thus, Consul is needed for service discovery for users to determine which port the Heron executor is serving the metrics out of.
+Every Heron executor will automatically register itself as a service with Consul given that there is a Consul cluster running. The port Heron will be serving metrics will be registered with Consul.
+
+The service will be registered with the name with the following format:
+
+```yaml
+metrics-heron-<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+Each heron executor registered with Consul will be tagged with
+
+```yaml
+<TOPOLOGY_NAME>-<CONTAINER_INDEX>
+```
+
+To add additional tags, please add specify them in a comma delimited list via
+
+```yaml
+heron.nomad.metrics.service.additional.tags
+```
+
+in `scheduler.yaml`. For example:
+
+```yaml
+heron.nomad.metrics.service.additional.tags: "prometheus,metrics,heron"
+```
+
+Users can then configure Prometheus to scrape metrics for each Heron executor based on these tags
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services