You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Brian Brazil <br...@boxever.com> on 2015/07/01 16:54:19 UTC
Using Prometheus with Aurora
Hi,
I've added support to Prometheus to use Twitter serversets for service
discovery, and we're now got this deployed in production. A key point is
that you can configure it to look at a tree of ZNodes in Zookeeper, so that
it can automatically pick up any new jobs announced by the Aurora executor
with no additional configuration required.
The scrape config we have in ansible for this is:
- job_name: "aurora_job"
serverset_sd_configs: - servers:{% for host in
groups["zookeeper"] %} - "{{ host }}:2181"{% endfor %}
paths: - "/aurora/serversets" relabel_configs: -
source_labels: [ __meta_serverset_path ] regex:
'^/aurora/serversets/[^/]+/[^/]+/([^/]+)/.*' target_label: job
replacement: "${1}" - source_labels: [ __address__ ]
regex: '(ip-\d+-\d+-\d+-\d+)\..*:(\d+)' target_label:
__address__ replacement: "${1}:${2}"
The relabel of the addresses is as our mesos slaves have the externally
visible host names so they work nicely in the browser, but which don't
actually exist in DNS so need some munging.
Note that you'll need the Prometheus at head to use this, and that this
should also work for non-Aurora created serversets such as if you're using
Finagle.
Relevant docs:
http://aurora.apache.org/documentation/latest/configuration-reference/#announcer-objects
http://prometheus.io/docs/operating/configuration/#zookeeper-serverset-sd-configurations-serverset_sd_config
Brian
Re: Using Prometheus with Aurora
Posted by Bill Farner <wf...@apache.org>.
Great work, Brian! This is awesome!
-=Bill
On Wed, Jul 1, 2015 at 7:54 AM, Brian Brazil <br...@boxever.com>
wrote:
> Hi,
>
> I've added support to Prometheus to use Twitter serversets for service
> discovery, and we're now got this deployed in production. A key point is
> that you can configure it to look at a tree of ZNodes in Zookeeper, so that
> it can automatically pick up any new jobs announced by the Aurora executor
> with no additional configuration required.
>
>
> The scrape config we have in ansible for this is:
>
> - job_name: "aurora_job"
> serverset_sd_configs: - servers:{% for host in
> groups["zookeeper"] %} - "{{ host }}:2181"{% endfor %}
> paths: - "/aurora/serversets" relabel_configs: -
> source_labels: [ __meta_serverset_path ] regex:
> '^/aurora/serversets/[^/]+/[^/]+/([^/]+)/.*' target_label: job
> replacement: "${1}" - source_labels: [ __address__ ]
> regex: '(ip-\d+-\d+-\d+-\d+)\..*:(\d+)' target_label:
> __address__ replacement: "${1}:${2}"
>
>
>
> The relabel of the addresses is as our mesos slaves have the externally
> visible host names so they work nicely in the browser, but which don't
> actually exist in DNS so need some munging.
>
> Note that you'll need the Prometheus at head to use this, and that this
> should also work for non-Aurora created serversets such as if you're using
> Finagle.
>
> Relevant docs:
>
> http://aurora.apache.org/documentation/latest/configuration-reference/#announcer-objects
>
>
> http://prometheus.io/docs/operating/configuration/#zookeeper-serverset-sd-configurations-serverset_sd_config
>
> Brian
>
Re: Using Prometheus with Aurora
Posted by Brian Brazil <br...@boxever.com>.
On 1 July 2015 at 18:06, Erb, Stephan <St...@blue-yonder.com> wrote:
> Hi Brian,
>
> that will come in handy. Thanks for that!
>
> I have thought about using prometheus together with the Mesos and Aurora
> exporters. Your feature would even allow us to start both exports on Aurora
> and let Prometheus discover them automatically. However, it might be a bit
> risky to use Aurora/Mesos to host its own monitoring solution...
>
> How have you organized the deployment of your Prometheus components?
>
Prometheus as-is is best considered a storage system in Mesos terms, so
isn't really suitable for putting on Mesos unless you've a quite short
retention period (hours?).
We're currently running Prometheus outside of Mesos via ansible+monit.
When I get around to exporting metrics from the mesos, that'll also be via
ansible+monit on each machine as that's cluster infrastructure and running
it on mesos itself would just complicate things. I'll likely end up using
EC2 service discovery for that (once it exists).
Brian
>
> Regards,
> Stephan
> ________________________________________
> From: Brian Brazil <br...@boxever.com>
> Sent: Wednesday, July 1, 2015 4:54 PM
> To: dev@aurora.apache.org; prometheus-developers
> Subject: Using Prometheus with Aurora
>
> Hi,
>
> I've added support to Prometheus to use Twitter serversets for service
> discovery, and we're now got this deployed in production. A key point is
> that you can configure it to look at a tree of ZNodes in Zookeeper, so that
> it can automatically pick up any new jobs announced by the Aurora executor
> with no additional configuration required.
>
>
> The scrape config we have in ansible for this is:
>
> - job_name: "aurora_job"
> serverset_sd_configs: - servers:{% for host in
> groups["zookeeper"] %} - "{{ host }}:2181"{% endfor %}
> paths: - "/aurora/serversets" relabel_configs: -
> source_labels: [ __meta_serverset_path ] regex:
> '^/aurora/serversets/[^/]+/[^/]+/([^/]+)/.*' target_label: job
> replacement: "${1}" - source_labels: [ __address__ ]
> regex: '(ip-\d+-\d+-\d+-\d+)\..*:(\d+)' target_label:
> __address__ replacement: "${1}:${2}"
>
>
>
> The relabel of the addresses is as our mesos slaves have the externally
> visible host names so they work nicely in the browser, but which don't
> actually exist in DNS so need some munging.
>
> Note that you'll need the Prometheus at head to use this, and that this
> should also work for non-Aurora created serversets such as if you're using
> Finagle.
>
> Relevant docs:
>
> http://aurora.apache.org/documentation/latest/configuration-reference/#announcer-objects
>
>
> http://prometheus.io/docs/operating/configuration/#zookeeper-serverset-sd-configurations-serverset_sd_config
>
> Brian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-developers+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
Re: Using Prometheus with Aurora
Posted by "Erb, Stephan" <St...@blue-yonder.com>.
Hi Brian,
that will come in handy. Thanks for that!
I have thought about using prometheus together with the Mesos and Aurora exporters. Your feature would even allow us to start both exports on Aurora and let Prometheus discover them automatically. However, it might be a bit risky to use Aurora/Mesos to host its own monitoring solution...
How have you organized the deployment of your Prometheus components?
Regards,
Stephan
________________________________________
From: Brian Brazil <br...@boxever.com>
Sent: Wednesday, July 1, 2015 4:54 PM
To: dev@aurora.apache.org; prometheus-developers
Subject: Using Prometheus with Aurora
Hi,
I've added support to Prometheus to use Twitter serversets for service
discovery, and we're now got this deployed in production. A key point is
that you can configure it to look at a tree of ZNodes in Zookeeper, so that
it can automatically pick up any new jobs announced by the Aurora executor
with no additional configuration required.
The scrape config we have in ansible for this is:
- job_name: "aurora_job"
serverset_sd_configs: - servers:{% for host in
groups["zookeeper"] %} - "{{ host }}:2181"{% endfor %}
paths: - "/aurora/serversets" relabel_configs: -
source_labels: [ __meta_serverset_path ] regex:
'^/aurora/serversets/[^/]+/[^/]+/([^/]+)/.*' target_label: job
replacement: "${1}" - source_labels: [ __address__ ]
regex: '(ip-\d+-\d+-\d+-\d+)\..*:(\d+)' target_label:
__address__ replacement: "${1}:${2}"
The relabel of the addresses is as our mesos slaves have the externally
visible host names so they work nicely in the browser, but which don't
actually exist in DNS so need some munging.
Note that you'll need the Prometheus at head to use this, and that this
should also work for non-Aurora created serversets such as if you're using
Finagle.
Relevant docs:
http://aurora.apache.org/documentation/latest/configuration-reference/#announcer-objects
http://prometheus.io/docs/operating/configuration/#zookeeper-serverset-sd-configurations-serverset_sd_config
Brian