You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Ali Nazemian <al...@gmail.com> on 2017/05/30 09:49:23 UTC
Metron components connectivity
Hi all,
For deploying Metron in production, we need to specify all of the port and
protocols connectivities. I was wondering how Metron components connected
to each other. Is there any document available regarding the ports and
connectivities of Metron components?
Regards,
Ali
Re: Metron components connectivity
Posted by Ali Nazemian <al...@gmail.com>.
We have already provided the list of generic HDP/HDF components and their
corresponding port/protocols which was one of the most painful works we
have ever done! However, I am struggling to understand those connectivities
from the Metron point of view. For example, how would be zookeeper
connectivities in Metron? What about Metron Management-UI, Alert-UI, Rest
API and Workflow API? Storm, Kafka and Elasticsearch? It would be great if
you can share what you have got already.
Cheers,
Ali
On Tue, May 30, 2017 at 8:10 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> I did this about 6 months ago but eventually gave up and allowed all ports
> between cluster nodes. I wasn't able to find a single source for this
> information - I used the HDP documentation as well as context I had from
> Metron and a bit of trial and error. That said, I probably only spent a
> couple of hours on it before giving up.
>
> I would be happy to share what I have, or help maintain something more
> complete if it exists somewhere that I'm not aware of.
>
> Jon
>
> On Tue, May 30, 2017, 5:49 AM Ali Nazemian <al...@gmail.com> wrote:
>
>> Hi all,
>>
>> For deploying Metron in production, we need to specify all of the port
>> and protocols connectivities. I was wondering how Metron components
>> connected to each other. Is there any document available regarding the
>> ports and connectivities of Metron components?
>>
>> Regards,
>> Ali
>>
> --
>
> Jon
>
--
A.Nazemian
Re: Metron components connectivity
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I did this about 6 months ago but eventually gave up and allowed all ports
between cluster nodes. I wasn't able to find a single source for this
information - I used the HDP documentation as well as context I had from
Metron and a bit of trial and error. That said, I probably only spent a
couple of hours on it before giving up.
I would be happy to share what I have, or help maintain something more
complete if it exists somewhere that I'm not aware of.
Jon
On Tue, May 30, 2017, 5:49 AM Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
> For deploying Metron in production, we need to specify all of the port and
> protocols connectivities. I was wondering how Metron components connected
> to each other. Is there any document available regarding the ports and
> connectivities of Metron components?
>
> Regards,
> Ali
>
--
Jon
Re: Metron components connectivity
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Yeah, those rules were for a working Metron cluster on 0.3.0 I believe.
Are you looking for a breakdown of rules per component? That's probably
the most sane way to approach this because components could be split in
different configurations.
I think that the blueprint approach with some manipulation may be the way
to go. I wonder if the Ambari team is looking at this sort of use case at
all. That said, I think that assumes all of the components are installed
via Ambari like the UI, etc.
Summary - it looks like this doesn't necessarily exist right now, but I
think it should. I would love to have some firewall rules better than a
hard shell with a gooey center.
Jon
On Wed, May 31, 2017, 4:32 AM Ali Nazemian <al...@gmail.com> wrote:
> Hi Jon,
>
> Thanks for the rules you have shared. Actually, my concern is the Metron
> use case parts and the corresponding components as follows. I was wondering
> how the connectivity works for them.
>
> -Metron Management UI
> -Metron Rest API
> -Metron Workflow Rest API
> -Metron Alert-UI
> -MySQL
> -HBase
> -Zookeeper
> -Storm Supervisor
> -Kafka broker
> -Elasticsearch Data Nodes
> -Kibana
>
> Cheers,
> Ali
>
>
> On Wed, May 31, 2017 at 2:54 AM, Matt Foley <ma...@apache.org> wrote:
>
>> ➢ It is a shame there isn’t a reporting tool for ambari, that can query
>> all the configurations in the database and report out ports and hosts…….
>>
>> There is. If you configure a “happy” cluster with Ambari, then tell
>> Ambari to generate a blueprint, that will tell you the actual as-built
>> values for all configuration parameters controlled by Ambari. Of course
>> there’s a lot of other stuff too, but the full config is there.
>>
>> Blueprint docs are at
>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
>> The API to export the current blueprint is at
>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step1:CreateBlueprint
>> The structure of the blueprint and config elements are in
>> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-BlueprintDetails
>>
>> Cheers,
>>
>>
>> --Matt
>>
>> From: Otto Fowler <ot...@gmail.com>
>> Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
>> Date: Tuesday, May 30, 2017 at 6:17 AM
>> To: Alex McLintock <al...@alephant.co.uk>, "user@metron.apache.org" <
>> user@metron.apache.org>
>> Subject: Re: Metron components connectivity
>>
>> It is a shame there isn’t a reporting tool for ambari, that can query all
>> the configurations in the database and report out ports and hosts…….
>>
>>
>> On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
>> I have attempted to do this for just HDP and it is rather difficult. One
>> of the main problems is that any documentation will point you towards
>> default ports for Hadoop - which are not always the ports chosen by
>> Hortonworks for HDP. You probably need to look mostly at the ports
>> specified in your config files.
>>
>> I would suggest that you treat all HDP nodes as able to talk to each
>> other across all ports - but limit anything which talks to those nodes.
>> That is a lot easier.
>>
>>
>> On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
>> Hi all,
>>
>> For deploying Metron in production, we need to specify all of the port
>> and protocols connectivities. I was wondering how Metron components
>> connected to each other. Is there any document available regarding the
>> ports and connectivities of Metron components?
>>
>> Regards,
>> Ali
>>
>>
>>
>>
>> --
>> Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to
>> win £50
>>
>>
>>
>>
>
>
> --
> A.Nazemian
>
--
Jon
Re: Metron components connectivity
Posted by Ali Nazemian <al...@gmail.com>.
Hi Jon,
Thanks for the rules you have shared. Actually, my concern is the Metron
use case parts and the corresponding components as follows. I was wondering
how the connectivity works for them.
-Metron Management UI
-Metron Rest API
-Metron Workflow Rest API
-Metron Alert-UI
-MySQL
-HBase
-Zookeeper
-Storm Supervisor
-Kafka broker
-Elasticsearch Data Nodes
-Kibana
Cheers,
Ali
On Wed, May 31, 2017 at 2:54 AM, Matt Foley <ma...@apache.org> wrote:
> ➢ It is a shame there isn’t a reporting tool for ambari, that can query
> all the configurations in the database and report out ports and hosts…….
>
> There is. If you configure a “happy” cluster with Ambari, then tell
> Ambari to generate a blueprint, that will tell you the actual as-built
> values for all configuration parameters controlled by Ambari. Of course
> there’s a lot of other stuff too, but the full config is there.
>
> Blueprint docs are at https://cwiki.apache.org/confluence/display/AMBARI/
> Blueprints
> The API to export the current blueprint is at https://cwiki.apache.org/
> confluence/display/AMBARI/Blueprints#Blueprints-Step1:CreateBlueprint
> The structure of the blueprint and config elements are in
> https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-
> BlueprintDetails
>
> Cheers,
>
>
> --Matt
>
> From: Otto Fowler <ot...@gmail.com>
> Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
> Date: Tuesday, May 30, 2017 at 6:17 AM
> To: Alex McLintock <al...@alephant.co.uk>, "user@metron.apache.org" <
> user@metron.apache.org>
> Subject: Re: Metron components connectivity
>
> It is a shame there isn’t a reporting tool for ambari, that can query all
> the configurations in the database and report out ports and hosts…….
>
>
> On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
> I have attempted to do this for just HDP and it is rather difficult. One
> of the main problems is that any documentation will point you towards
> default ports for Hadoop - which are not always the ports chosen by
> Hortonworks for HDP. You probably need to look mostly at the ports
> specified in your config files.
>
> I would suggest that you treat all HDP nodes as able to talk to each other
> across all ports - but limit anything which talks to those nodes. That is a
> lot easier.
>
>
> On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
> For deploying Metron in production, we need to specify all of the port and
> protocols connectivities. I was wondering how Metron components connected
> to each other. Is there any document available regarding the ports and
> connectivities of Metron components?
>
> Regards,
> Ali
>
>
>
>
> --
> Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to
> win £50
>
>
>
>
--
A.Nazemian
Re: Metron components connectivity
Posted by Matt Foley <ma...@apache.org>.
➢ It is a shame there isn’t a reporting tool for ambari, that can query all the configurations in the database and report out ports and hosts…….
There is. If you configure a “happy” cluster with Ambari, then tell Ambari to generate a blueprint, that will tell you the actual as-built values for all configuration parameters controlled by Ambari. Of course there’s a lot of other stuff too, but the full config is there.
Blueprint docs are at https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
The API to export the current blueprint is at https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step1:CreateBlueprint
The structure of the blueprint and config elements are in https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-BlueprintDetails
Cheers,
--Matt
From: Otto Fowler <ot...@gmail.com>
Reply-To: "user@metron.apache.org" <us...@metron.apache.org>
Date: Tuesday, May 30, 2017 at 6:17 AM
To: Alex McLintock <al...@alephant.co.uk>, "user@metron.apache.org" <us...@metron.apache.org>
Subject: Re: Metron components connectivity
It is a shame there isn’t a reporting tool for ambari, that can query all the configurations in the database and report out ports and hosts…….
On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
I have attempted to do this for just HDP and it is rather difficult. One of the main problems is that any documentation will point you towards default ports for Hadoop - which are not always the ports chosen by Hortonworks for HDP. You probably need to look mostly at the ports specified in your config files.
I would suggest that you treat all HDP nodes as able to talk to each other across all ports - but limit anything which talks to those nodes. That is a lot easier.
On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
Hi all,
For deploying Metron in production, we need to specify all of the port and protocols connectivities. I was wondering how Metron components connected to each other. Is there any document available regarding the ports and connectivities of Metron components?
Regards,
Ali
--
Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to win £50
Re: Metron components connectivity
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Here's what I used a while ago in the form of an iptables file, cleansed
slightly. As far as I recall I think this was before I customized any of
the ports. Sorry for the poor comments, I was pretty tired of the whole
process towards the end and became less and less verbose.
# Allow outbound proxy traffic
-A custom-rules -p tcp -m tcp --dport 12345 -j ACCEPT
# Allow remote SSH
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 22 -j ACCEPT
# Allow SSH for file transfer
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
# Allow intra-cluster SSH
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 22 -j ACCEPT
# Allow Ambari server access
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8080 -j ACCEPT
# Allow Ambari access from subnet
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 8080 -j ACCEPT
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 8440 -j ACCEPT
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 8441 -j ACCEPT
# Allow Ambari Agent traffic
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8670 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8441 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8440 -j ACCEPT
# Various cluster management
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8020 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 6188 -j ACCEPT
# Tons of internal communication
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50070 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50470 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 9000 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50075 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50475 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50010 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50020 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50090 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50030 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8021 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50060 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 51111 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 10000 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 9083 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 60000 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 60010 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 60020 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 60030 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 2888 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 3888 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 2181 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 50111 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8660 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 61 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 62 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 63 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8651 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 3306 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8025 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 16000 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 6667 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 8010 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 16020 -j ACCEPT
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 60414 -j ACCEPT
# Access to the web UIs
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 80 -j ACCEPT
-A custom-rules -s x.x.x.x/x -p tcp -m tcp --dport 8088 -j ACCEPT
# Allow Kafka traffic
-A custom-rules -s x.x.x.x -p tcp -m tcp --dport 9092 -j ACCEPT
Jon
On Tue, May 30, 2017 at 10:09 AM Ali Nazemian <al...@gmail.com> wrote:
> I have provided the complete list regarding generic HDP/HDF ports, but I
> couldn't find the Metron specific ones...
>
> On Tue, May 30, 2017 at 11:17 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> It is a shame there isn’t a reporting tool for ambari, that can query all
>> the configurations in the database and report out ports and hosts…….
>>
>>
>> On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
>>
>> I have attempted to do this for just HDP and it is rather difficult. One
>> of the main problems is that any documentation will point you towards
>> default ports for Hadoop - which are not always the ports chosen by
>> Hortonworks for HDP. You probably need to look mostly at the ports
>> specified in your config files.
>>
>> I would suggest that you treat all HDP nodes as able to talk to each
>> other across all ports - but limit anything which talks to those nodes.
>> That is a lot easier.
>>
>>
>> On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> For deploying Metron in production, we need to specify all of the port
>>> and protocols connectivities. I was wondering how Metron components
>>> connected to each other. Is there any document available regarding the
>>> ports and connectivities of Metron components?
>>>
>>> Regards,
>>> Ali
>>>
>>
>>
>>
>> --
>> Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to
>> win £50
>>
>>
>
>
> --
> A.Nazemian
>
--
Jon
Re: Metron components connectivity
Posted by Ali Nazemian <al...@gmail.com>.
I have provided the complete list regarding generic HDP/HDF ports, but I
couldn't find the Metron specific ones...
On Tue, May 30, 2017 at 11:17 PM, Otto Fowler <ot...@gmail.com>
wrote:
> It is a shame there isn’t a reporting tool for ambari, that can query all
> the configurations in the database and report out ports and hosts…….
>
>
> On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
>
> I have attempted to do this for just HDP and it is rather difficult. One
> of the main problems is that any documentation will point you towards
> default ports for Hadoop - which are not always the ports chosen by
> Hortonworks for HDP. You probably need to look mostly at the ports
> specified in your config files.
>
> I would suggest that you treat all HDP nodes as able to talk to each other
> across all ports - but limit anything which talks to those nodes. That is a
> lot easier.
>
>
> On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
>
>> Hi all,
>>
>> For deploying Metron in production, we need to specify all of the port
>> and protocols connectivities. I was wondering how Metron components
>> connected to each other. Is there any document available regarding the
>> ports and connectivities of Metron components?
>>
>> Regards,
>> Ali
>>
>
>
>
> --
> Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to
> win £50
>
>
--
A.Nazemian
Re: Metron components connectivity
Posted by Otto Fowler <ot...@gmail.com>.
It is a shame there isn’t a reporting tool for ambari, that can query all the configurations in the database and report out ports and hosts…….
On May 30, 2017 at 08:34:42, Alex McLintock (alex@alephant.co.uk) wrote:
I have attempted to do this for just HDP and it is rather difficult. One of the main problems is that any documentation will point you towards default ports for Hadoop - which are not always the ports chosen by Hortonworks for HDP. You probably need to look mostly at the ports specified in your config files.
I would suggest that you treat all HDP nodes as able to talk to each other across all ports - but limit anything which talks to those nodes. That is a lot easier.
On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
Hi all,
For deploying Metron in production, we need to specify all of the port and protocols connectivities. I was wondering how Metron components connected to each other. Is there any document available regarding the ports and connectivities of Metron components?
Regards,
Ali
--
Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to win £50
Re: Metron components connectivity
Posted by Alex McLintock <al...@alephant.co.uk>.
I have attempted to do this for just HDP and it is rather difficult. One of
the main problems is that any documentation will point you towards default
ports for Hadoop - which are not always the ports chosen by Hortonworks for
HDP. You probably need to look mostly at the ports specified in your config
files.
I would suggest that you treat all HDP nodes as able to talk to each other
across all ports - but limit anything which talks to those nodes. That is a
lot easier.
On 30 May 2017 at 10:49, Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
> For deploying Metron in production, we need to specify all of the port and
> protocols connectivities. I was wondering how Metron components connected
> to each other. Is there any document available regarding the ports and
> connectivities of Metron components?
>
> Regards,
> Ali
>
--
Want the best UK suppliers? Visit http://SocialConsumer.co.uk/ Draw to win
£50