You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Ali Nazemian <al...@gmail.com> on 2017/10/05 08:53:56 UTC
Ambari metrics monitor-collector over the WAN
Hi all,
I am investigating to have a huge cluster of Nifi instances distributed
across the world. I have been told that Metrics Monitor and Metrics
Collector are using a very chatty protocol, so Ambari Metrics Collector may
face some serious performance issues regarding receiving telemetry
information from lots of Metrics Monitor Agents. I have two concerns
regarding that as follows.
1- How does it scale for thousands of agents?
2- Can I use it over WAN probably by securing that in a two-way SSL way?
Regards,
Ali
Re: Ambari metrics monitor-collector over the WAN
Posted by Aravindan Vijayan <av...@hortonworks.com>.
Hello,
It is one way SSL. The metric monitors (clients) verifies the metrics collector’s certificate and then ‘POST’s the metrics.
--
Thanks and Regards,
Aravindan Vijayan
From: Ali Nazemian <al...@gmail.com>
Reply-To: "user@ambari.apache.org" <us...@ambari.apache.org>
Date: Thursday, October 5, 2017 at 2:48 PM
To: "user@ambari.apache.org" <us...@ambari.apache.org>
Cc: "qinliu5678@gmail.com" <qi...@gmail.com>
Subject: Re: Ambari metrics monitor-collector over the WAN
Perfect. Thank you very much. Regarding https, is it 2-way or 1 way?
On 6 Oct. 2017 04:30, "Siddharth Wagle" <sw...@hortonworks.com>> wrote:
1- How does it scale for thousands of agents?
With Ambari 2.5.2 you can launch a 2 node collector in distributed mode and select the placement of the node based on geographical location of the agents.
Unfortunately, we shard based on hostname to select from the monitor, which collector to write to.
Presently this is not customizable however, if you need it to be feel free to open a Jira, it is would be a straightforward fix required. However, I doubt you would need to do this because all sinks locally aggregate metrics and write only after 1 minute of data is collected.
https://issues.apache.org/jira/browse/AMBARI-15901
Regarding volume of writes: We have tested the system with 2 node collector upto 2K nodes and we already know of production systems of up to 1500 node working with AMS.
Since the collection model is push based the bottleneck needs to be solved on the collector and therefore in 2.5.1+ version of Ambari you can add additional collector node to handle extra load.
Note: In 3.0 we are doing quite a few things to address performance by de-centralizing aggregations and reducing memory bloat on RegionServers etc, but that will be ready later in the year.
2- Can I use it over WAN probably by securing that in a two-way SSL way?
Regarding security: We already support https for AMS - https://issues.apache.org/jira/browse/AMBARI-15050
The SPNEGO based authentication work is still not complete: We have a quite a few patches already in trunk but haven't looked at release readiness for these.
Possible Qin Liu can shed more light on this.
BR,
Sid
________________________________
From: Ali Nazemian <al...@gmail.com>>
Sent: Thursday, October 5, 2017 1:53 AM
To: user@ambari.apache.org<ma...@ambari.apache.org>
Subject: Ambari metrics monitor-collector over the WAN
Hi all,
I am investigating to have a huge cluster of Nifi instances distributed across the world. I have been told that Metrics Monitor and Metrics Collector are using a very chatty protocol, so Ambari Metrics Collector may face some serious performance issues regarding receiving telemetry information from lots of Metrics Monitor Agents. I have two concerns regarding that as follows.
1- How does it scale for thousands of agents?
2- Can I use it over WAN probably by securing that in a two-way SSL way?
Regards,
Ali
Re: Ambari metrics monitor-collector over the WAN
Posted by Ali Nazemian <al...@gmail.com>.
Perfect. Thank you very much. Regarding https, is it 2-way or 1 way?
On 6 Oct. 2017 04:30, "Siddharth Wagle" <sw...@hortonworks.com> wrote:
> 1- How does it scale for thousands of agents?
>
>
> With Ambari 2.5.2 you can launch a 2 node collector in distributed mode
> and select the placement of the node based on geographical location of the
> agents.
>
> Unfortunately, we shard based on hostname to select from the monitor,
> which collector to write to.
>
> Presently this is not customizable however, if you need it to be feel free
> to open a Jira, it is would be a straightforward fix required. However, I
> doubt you would need to do this because all sinks locally aggregate metrics
> and write only after 1 minute of data is collected.
>
>
> https://issues.apache.org/jira/browse/AMBARI-15901
>
>
> Regarding volume of writes: We have tested the system with 2 node
> collector upto 2K nodes and we already know of production systems of
> up to 1500 node working with AMS.
>
> Since the collection model is push based the bottleneck needs to be solved
> on the collector and therefore in 2.5.1+ version of Ambari you can add
> additional collector node to handle extra load.
>
>
> *Note*: In 3.0 we are doing quite a few things to address performance by
> de-centralizing aggregations and reducing memory bloat on RegionServers
> etc, but that will be ready later in the year.
>
>
> 2- Can I use it over WAN probably by securing that in a two-way SSL way?
>
>
> Regarding security: We already support https for AMS -
> https://issues.apache.org/jira/browse/AMBARI-15050
>
>
> The SPNEGO based authentication work is still not complete: We have a
> quite a few patches already in trunk but haven't looked at release
> readiness for these.
>
> Possible Qin Liu can shed more light on this.
>
>
> BR,
>
> Sid
>
>
> ------------------------------
> *From:* Ali Nazemian <al...@gmail.com>
> *Sent:* Thursday, October 5, 2017 1:53 AM
> *To:* user@ambari.apache.org
> *Subject:* Ambari metrics monitor-collector over the WAN
>
> Hi all,
>
> I am investigating to have a huge cluster of Nifi instances distributed
> across the world. I have been told that Metrics Monitor and Metrics
> Collector are using a very chatty protocol, so Ambari Metrics Collector may
> face some serious performance issues regarding receiving telemetry
> information from lots of Metrics Monitor Agents. I have two concerns
> regarding that as follows.
>
> 1- How does it scale for thousands of agents?
>
> 2- Can I use it over WAN probably by securing that in a two-way SSL way?
>
>
> Regards,
>
> Ali
>
Re: Ambari metrics monitor-collector over the WAN
Posted by Siddharth Wagle <sw...@hortonworks.com>.
1- How does it scale for thousands of agents?
With Ambari 2.5.2 you can launch a 2 node collector in distributed mode and select the placement of the node based on geographical location of the agents.
Unfortunately, we shard based on hostname to select from the monitor, which collector to write to.
Presently this is not customizable however, if you need it to be feel free to open a Jira, it is would be a straightforward fix required. However, I doubt you would need to do this because all sinks locally aggregate metrics and write only after 1 minute of data is collected.
https://issues.apache.org/jira/browse/AMBARI-15901
Regarding volume of writes: We have tested the system with 2 node collector upto 2K nodes and we already know of production systems of up to 1500 node working with AMS.
Since the collection model is push based the bottleneck needs to be solved on the collector and therefore in 2.5.1+ version of Ambari you can add additional collector node to handle extra load.
Note: In 3.0 we are doing quite a few things to address performance by de-centralizing aggregations and reducing memory bloat on RegionServers etc, but that will be ready later in the year.
2- Can I use it over WAN probably by securing that in a two-way SSL way?
Regarding security: We already support https for AMS - https://issues.apache.org/jira/browse/AMBARI-15050
The SPNEGO based authentication work is still not complete: We have a quite a few patches already in trunk but haven't looked at release readiness for these.
Possible Qin Liu can shed more light on this.
BR,
Sid
________________________________
From: Ali Nazemian <al...@gmail.com>
Sent: Thursday, October 5, 2017 1:53 AM
To: user@ambari.apache.org
Subject: Ambari metrics monitor-collector over the WAN
Hi all,
I am investigating to have a huge cluster of Nifi instances distributed across the world. I have been told that Metrics Monitor and Metrics Collector are using a very chatty protocol, so Ambari Metrics Collector may face some serious performance issues regarding receiving telemetry information from lots of Metrics Monitor Agents. I have two concerns regarding that as follows.
1- How does it scale for thousands of agents?
2- Can I use it over WAN probably by securing that in a two-way SSL way?
Regards,
Ali