You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Kwan Chong Tan <kc...@gmail.com> on 2016/07/12 00:52:46 UTC
Metron Single Node AWS Deployment
Hi all,
I was wondering if its possible to deploy Metron on a single node (or fewer
node) configuration on AWS. I managed to get the full 10 node cluster stood
up on AWS but for exploration purposes I don't need or want to maintain
such a large cluster. I tried modifying the number of hosts in the
playbook.yml file and combining roles but the installation fails at various
steps.
Best regards,
KC
Re: Metron Single Node AWS Deployment
Posted by Nick Allen <ni...@nickallen.org>.
I am also able to replicate the problem that you are experiencing with the
Kibana map widget in Metron's default dashboard. I opened a JIRA so that
we can track it. It appears that the Kibana project is working on a hotfix
based on changes made by MapQuest.
https://issues.apache.org/jira/browse/METRON-314
On Tue, Jul 12, 2016 at 5:52 AM, Kwan Chong Tan <kc...@gmail.com> wrote:
> Hi Nick,
>
> I managed to get it working by:
>
> 1. changing conf/defaults.yml cluster_type: 'small_cluster' to
> 'single_node_vm'
> 2. changing ambari_config/vars/single_node_vm.yml groups: hosts: "{{
> groups.ambari_master }}"
>
> On a separate note, I constantly get an error at the snort: Configure home
> network task and have to manually execute the step. Not sure if it's due to
> a configuration error on my end or an issue with the ansible scripts.
>
> TASK [snort : Configure home network]
> ******************************************
> fatal: [ec2-54-213-253-12.us-west-2.compute.amazonaws.com]: FAILED! =>
> {"failed": true, "msg": "ERROR! ERROR! 'dict object' has no attribute
> u'ansible_tap0'"}
>
> Finally, I also noticed that the MapQuest tile support seems to have been
> discontinued?
>
> [image: Inline image 1]
>
> Thanks,
> KC
>
> On Tue, Jul 12, 2016 at 11:09 AM, Kwan Chong Tan <kc...@gmail.com> wrote:
>
>> Hi Nick,
>>
>> So the installation failed at the deploy cluster with Ambari step. Below
>> is a link to the tail end of the log. Thanks!
>>
>> https://gist.github.com/thufirtan/65362dd91376779ca586d602ef8efc1c
>>
>> On Tue, Jul 12, 2016 at 10:47 AM, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> I've never done it, but in theory it should work. You would just need to
>>> be cautious of the amount of memory and CPU available on your host.
>>>
>>> There was some level of tuning done to get the entire stack to run on a
>>> Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
>>> settings appropriate for a small host.
>>>
>>> There is a setting that you could potentially change to pull in those
>>> "small" settings, but it's escaping me at the moment.
>>> On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>
>>>> Thanks Nick. I'll try this again and post the logs. What's the lowest /
>>>> minimum number of nodes I can go to? Can I just use one node and assign all
>>>> the roles to one host?
>>>>
>>>> e.g. in playbook.yml
>>>>
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>>>>
>>>>
>>>> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org>
>>>> wrote:
>>>>
>>>>> That should work. Can you send the modifications that you made? Also
>>>>> post your ansible.log somewhere and I'll take a look at the errors that you
>>>>> encountered.
>>>>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I was wondering if its possible to deploy Metron on a single node (or
>>>>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>>>>> stood up on AWS but for exploration purposes I don't need or want to
>>>>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>>>>> playbook.yml file and combining roles but the installation fails at various
>>>>>> steps.
>>>>>>
>>>>>> Best regards,
>>>>>> KC
>>>>>>
>>>>>
>>>>
>>
>
--
Nick Allen <ni...@nickallen.org>
Re: Metron Single Node AWS Deployment
Posted by Kwan Chong Tan <kc...@gmail.com>.
Hi Nick,
I managed to get it working by:
1. changing conf/defaults.yml cluster_type: 'small_cluster' to
'single_node_vm'
2. changing ambari_config/vars/single_node_vm.yml groups: hosts: "{{
groups.ambari_master }}"
On a separate note, I constantly get an error at the snort: Configure home
network task and have to manually execute the step. Not sure if it's due to
a configuration error on my end or an issue with the ansible scripts.
TASK [snort : Configure home network]
******************************************
fatal: [ec2-54-213-253-12.us-west-2.compute.amazonaws.com]: FAILED! =>
{"failed": true, "msg": "ERROR! ERROR! 'dict object' has no attribute
u'ansible_tap0'"}
Finally, I also noticed that the MapQuest tile support seems to have been
discontinued?
[image: Inline image 1]
Thanks,
KC
On Tue, Jul 12, 2016 at 11:09 AM, Kwan Chong Tan <kc...@gmail.com> wrote:
> Hi Nick,
>
> So the installation failed at the deploy cluster with Ambari step. Below
> is a link to the tail end of the log. Thanks!
>
> https://gist.github.com/thufirtan/65362dd91376779ca586d602ef8efc1c
>
> On Tue, Jul 12, 2016 at 10:47 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> I've never done it, but in theory it should work. You would just need to
>> be cautious of the amount of memory and CPU available on your host.
>>
>> There was some level of tuning done to get the entire stack to run on a
>> Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
>> settings appropriate for a small host.
>>
>> There is a setting that you could potentially change to pull in those
>> "small" settings, but it's escaping me at the moment.
>> On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>
>>> Thanks Nick. I'll try this again and post the logs. What's the lowest /
>>> minimum number of nodes I can go to? Can I just use one node and assign all
>>> the roles to one host?
>>>
>>> e.g. in playbook.yml
>>>
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>>>
>>>
>>> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org> wrote:
>>>
>>>> That should work. Can you send the modifications that you made? Also
>>>> post your ansible.log somewhere and I'll take a look at the errors that you
>>>> encountered.
>>>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I was wondering if its possible to deploy Metron on a single node (or
>>>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>>>> stood up on AWS but for exploration purposes I don't need or want to
>>>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>>>> playbook.yml file and combining roles but the installation fails at various
>>>>> steps.
>>>>>
>>>>> Best regards,
>>>>> KC
>>>>>
>>>>
>>>
>
Re: Metron Single Node AWS Deployment
Posted by Kwan Chong Tan <kc...@gmail.com>.
Hi Nick,
Not sure if you missed my earlier email but I managed to get it working
after changing a few other parameters in the Ansible scripts:
1. changing conf/defaults.yml cluster_type: 'small_cluster' to
'single_node_vm'
2. changing ambari_config/vars/single_node_vm.yml groups: hosts: "{{
groups.ambari_master }}"
Thanks again for your assistance in the matter!
Best regards,
KC
On Tue, Jul 12, 2016 at 9:04 PM, Nick Allen <ni...@nickallen.org> wrote:
> Can you post the entire log? Or at least the last couple runs that you
> did.
>
> On Mon, Jul 11, 2016 at 11:09 PM, Kwan Chong Tan <kc...@gmail.com> wrote:
>
>> Hi Nick,
>>
>> So the installation failed at the deploy cluster with Ambari step. Below
>> is a link to the tail end of the log. Thanks!
>>
>> https://gist.github.com/thufirtan/65362dd91376779ca586d602ef8efc1c
>>
>> On Tue, Jul 12, 2016 at 10:47 AM, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> I've never done it, but in theory it should work. You would just need to
>>> be cautious of the amount of memory and CPU available on your host.
>>>
>>> There was some level of tuning done to get the entire stack to run on a
>>> Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
>>> settings appropriate for a small host.
>>>
>>> There is a setting that you could potentially change to pull in those
>>> "small" settings, but it's escaping me at the moment.
>>> On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>
>>>> Thanks Nick. I'll try this again and post the logs. What's the lowest /
>>>> minimum number of nodes I can go to? Can I just use one node and assign all
>>>> the roles to one host?
>>>>
>>>> e.g. in playbook.yml
>>>>
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>>>>
>>>>
>>>> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org>
>>>> wrote:
>>>>
>>>>> That should work. Can you send the modifications that you made? Also
>>>>> post your ansible.log somewhere and I'll take a look at the errors that you
>>>>> encountered.
>>>>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I was wondering if its possible to deploy Metron on a single node (or
>>>>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>>>>> stood up on AWS but for exploration purposes I don't need or want to
>>>>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>>>>> playbook.yml file and combining roles but the installation fails at various
>>>>>> steps.
>>>>>>
>>>>>> Best regards,
>>>>>> KC
>>>>>>
>>>>>
>>>>
>>
>
>
> --
> Nick Allen <ni...@nickallen.org>
>
Re: Metron Single Node AWS Deployment
Posted by Nick Allen <ni...@nickallen.org>.
Can you post the entire log? Or at least the last couple runs that you did.
On Mon, Jul 11, 2016 at 11:09 PM, Kwan Chong Tan <kc...@gmail.com> wrote:
> Hi Nick,
>
> So the installation failed at the deploy cluster with Ambari step. Below
> is a link to the tail end of the log. Thanks!
>
> https://gist.github.com/thufirtan/65362dd91376779ca586d602ef8efc1c
>
> On Tue, Jul 12, 2016 at 10:47 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> I've never done it, but in theory it should work. You would just need to
>> be cautious of the amount of memory and CPU available on your host.
>>
>> There was some level of tuning done to get the entire stack to run on a
>> Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
>> settings appropriate for a small host.
>>
>> There is a setting that you could potentially change to pull in those
>> "small" settings, but it's escaping me at the moment.
>> On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>
>>> Thanks Nick. I'll try this again and post the logs. What's the lowest /
>>> minimum number of nodes I can go to? Can I just use one node and assign all
>>> the roles to one host?
>>>
>>> e.g. in playbook.yml
>>>
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>>>
>>>
>>> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org> wrote:
>>>
>>>> That should work. Can you send the modifications that you made? Also
>>>> post your ansible.log somewhere and I'll take a look at the errors that you
>>>> encountered.
>>>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I was wondering if its possible to deploy Metron on a single node (or
>>>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>>>> stood up on AWS but for exploration purposes I don't need or want to
>>>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>>>> playbook.yml file and combining roles but the installation fails at various
>>>>> steps.
>>>>>
>>>>> Best regards,
>>>>> KC
>>>>>
>>>>
>>>
>
--
Nick Allen <ni...@nickallen.org>
Re: Metron Single Node AWS Deployment
Posted by Kwan Chong Tan <kc...@gmail.com>.
Hi Nick,
So the installation failed at the deploy cluster with Ambari step. Below is
a link to the tail end of the log. Thanks!
https://gist.github.com/thufirtan/65362dd91376779ca586d602ef8efc1c
On Tue, Jul 12, 2016 at 10:47 AM, Nick Allen <ni...@nickallen.org> wrote:
> I've never done it, but in theory it should work. You would just need to
> be cautious of the amount of memory and CPU available on your host.
>
> There was some level of tuning done to get the entire stack to run on a
> Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
> settings appropriate for a small host.
>
> There is a setting that you could potentially change to pull in those
> "small" settings, but it's escaping me at the moment.
> On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>
>> Thanks Nick. I'll try this again and post the logs. What's the lowest /
>> minimum number of nodes I can go to? Can I just use one node and assign all
>> the roles to one host?
>>
>> e.g. in playbook.yml
>>
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>>
>>
>> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> That should work. Can you send the modifications that you made? Also
>>> post your ansible.log somewhere and I'll take a look at the errors that you
>>> encountered.
>>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I was wondering if its possible to deploy Metron on a single node (or
>>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>>> stood up on AWS but for exploration purposes I don't need or want to
>>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>>> playbook.yml file and combining roles but the installation fails at various
>>>> steps.
>>>>
>>>> Best regards,
>>>> KC
>>>>
>>>
>>
Re: Metron Single Node AWS Deployment
Posted by Nick Allen <ni...@nickallen.org>.
I've never done it, but in theory it should work. You would just need to be
cautious of the amount of memory and CPU available on your host.
There was some level of tuning done to get the entire stack to run on a
Vagrant image with 8 GB of RAM. If you deploy via AWS then it will not have
settings appropriate for a small host.
There is a setting that you could potentially change to pull in those
"small" settings, but it's escaping me at the moment.
On Jul 11, 2016 10:37 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
> Thanks Nick. I'll try this again and post the logs. What's the lowest /
> minimum number of nodes I can go to? Can I just use one node and assign all
> the roles to one host?
>
> e.g. in playbook.yml
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
>
>
> On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> That should work. Can you send the modifications that you made? Also
>> post your ansible.log somewhere and I'll take a look at the errors that you
>> encountered.
>> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I was wondering if its possible to deploy Metron on a single node (or
>>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>>> stood up on AWS but for exploration purposes I don't need or want to
>>> maintain such a large cluster. I tried modifying the number of hosts in the
>>> playbook.yml file and combining roles but the installation fails at various
>>> steps.
>>>
>>> Best regards,
>>> KC
>>>
>>
>
Re: Metron Single Node AWS Deployment
Posted by Kwan Chong Tan <kc...@gmail.com>.
Thanks Nick. I'll try this again and post the logs. What's the lowest /
minimum number of nodes I can go to? Can I just use one node and assign all
the roles to one host?
e.g. in playbook.yml
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,metron_kafka_topics,metron_hbase_tables,metron,ec2,pcap_server,ambari_slave,web,mysql,search,enrichment
On Tue, Jul 12, 2016 at 10:27 AM, Nick Allen <ni...@nickallen.org> wrote:
> That should work. Can you send the modifications that you made? Also post
> your ansible.log somewhere and I'll take a look at the errors that you
> encountered.
> On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
>
>> Hi all,
>>
>> I was wondering if its possible to deploy Metron on a single node (or
>> fewer node) configuration on AWS. I managed to get the full 10 node cluster
>> stood up on AWS but for exploration purposes I don't need or want to
>> maintain such a large cluster. I tried modifying the number of hosts in the
>> playbook.yml file and combining roles but the installation fails at various
>> steps.
>>
>> Best regards,
>> KC
>>
>
Re: Metron Single Node AWS Deployment
Posted by Nick Allen <ni...@nickallen.org>.
That should work. Can you send the modifications that you made? Also post
your ansible.log somewhere and I'll take a look at the errors that you
encountered.
On Jul 11, 2016 8:53 PM, "Kwan Chong Tan" <kc...@gmail.com> wrote:
> Hi all,
>
> I was wondering if its possible to deploy Metron on a single node (or
> fewer node) configuration on AWS. I managed to get the full 10 node cluster
> stood up on AWS but for exploration purposes I don't need or want to
> maintain such a large cluster. I tried modifying the number of hosts in the
> playbook.yml file and combining roles but the installation fails at various
> steps.
>
> Best regards,
> KC
>