You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Laurens Vets <la...@daemon.be> on 2017/05/26 00:50:37 UTC
AWS single_node_vm
Is AWS single_node_vm still supported?
Re: AWS single_node_vm
Posted by Dima Kovalyov <Di...@sstech.us>.
I would say yes, at the ages of Metron 0.2.something I had successful
experience with installing it on single node Amazon using this playbook.
As Michael said, I did play with playbook.yml for that a bit, in the
past when I googled I found following article:
https://community.hortonworks.com/questions/44494/metron-single-node-aws-deployment.html
Still i had to do some more manual hacks, but in the end it payed-off.
Maybe there is more info out there about it.
Unfortunately the repo I was using long gone, but hey, you can do it :)
p.s. also checkout this thread:
https://mail-archives.apache.org/mod_mbox/incubator-metron-user/201607.mbox/%3CCAEJAfg1J5wNfugqXGCvjLLQc1_WTbj4L6pBeePx9GCuS+=au7Q@mail.gmail.com%3E
- Dima
On 05/26/2017 10:46 PM, David Lyle wrote:
> I'm gonna to venture a 'no' simply because I don't know what that is.
> Afaik, single_node_vm.yml was never used with AWS. What are you trying to
> do?
>
> -D...
>
>
> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be> wrote:
>
>> Is AWS single_node_vm still supported?
>>
Re: AWS single_node_vm
Posted by Laurens Vets <la...@daemon.be>.
Hi David,
That works, thanks! Now trying to get this down even more (4 sounds like
a nice number :p)
On 2017-05-30 17:05, David Lyle wrote:
> This one should do the trick for you.
>
> Due to how smal_cluster.yml is structured (designed for 10+ nodes)-
> you'll
> only get 2 data nodes, so you'll have to reduce replication to 2 if you
> don't want to see under-replicated block warnings.
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
> - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ambari_slave,web,ec2
>
> The reason the first one didn't work is because order matters. In the
> that
> example, the host created by
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>
> was ending up in 2 separate hostgroups, which is not allowed.
> Relocating
> that line solves the problem. I also relocated the web hosts (ES Master
> and
> Kibana) since there were fewer hosts.
>
>
> -D...
>
>
> On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> I put in what you said previously :)
>>
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,ec2
>>
>> On 2017-05-30 08:53, David Lyle wrote:
>>
>>> That's actually a different error than you posted before. That error
>>> is
>>> Ambari telling you that the blueprint that Ansible submitted is
>>> invalid.
>>> Likely a result of a required hostgroup not being defined. Can you
>>> (re?)
>>> post what you've put in amazon-ec2/playbook.yml?
>>>
>>> -D...
>>>
>>>
>>> On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <la...@daemon.be>
>>> wrote:
>>>
>>> Same problem as all the previous ones...
>>>>
>>>> TASK [ambari_config : Install python-requests]
>>>> *********************************
>>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] =>
>>>> {"attempts":
>>>> 1, "changed": false, "msg": "", "rc": 0, "results":
>>>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is
>>>> already
>>>> installed"]}
>>>>
>>>> TASK [ambari_config : check if ambari-server is up on
>>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] =>
>>>> {"changed":
>>>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex":
>>>> null,
>>>> "state": "started"}
>>>>
>>>> TASK [ambari_config : Deploy cluster with Ambari;
>>>> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>>> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED!
>>>> =>
>>>> {"changed": false, "failed": true, "msg": "Ambari client exception
>>>> occurred: Could not create cluster: request code 400,
>>>> request message {\n \"status\" : 400,\n \"message\" : \"Topology
>>>> validation failed: org.apache.ambari.server.topol
>>>> ogy.InvalidTopologyException:
>>>> The following hosts are mapped to multiple host groups: [
>>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that
>>>> host
>>>> names are converted to lowercase, case differences do not matter in
>>>> Ambari
>>>> deployments.\"\n}"}
>>>> to retry, use: --limit @/root/metron/metron-deploymen
>>>> t/amazon-ec2/playbook.retry
>>>>
>>>> PLAY RECAP
>>>> ************************************************************
>>>> *********
>>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47
>>>> changed=31
>>>> unreachable=0 failed=0
>>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48
>>>> changed=28
>>>> unreachable=0 failed=1
>>>> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17
>>>> changed=8
>>>> unreachable=0 failed=0
>>>> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>> unreachable=0 failed=0
>>>> localhost : ok=18 changed=13 unreachable=0
>>>> failed=0
>>>>
>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>
>>>>
>>>> On 2017-05-27 04:45, David Lyle wrote:
>>>>
>>>> For some reason, kibana_hosts is undefined, so the curl command that
>>>>> loads
>>>>> the Kibana dashboard doesn't contain the URL for Kibana. If you
>>>>> look at
>>>>> small_cluster.yml, you'll see that the Kibana should have been
>>>>> installed
>>>>> on
>>>>> the first host in the "web" hostgroup. Verify that you placed a
>>>>> host in
>>>>> that hostgroup (it's the 5th line in Mike's example). If you did,
>>>>> check
>>>>> that host on Ambari and verify that Kibana was installed, if not,
>>>>> you'll
>>>>> want to dig through the install log to find out why.
>>>>>
>>>>> Also- now that I've looked at the example carefully, I noticed that
>>>>> the
>>>>> Zeppelin hostgroup was duplicated. That's not a problem, because
>>>>> the 2nd
>>>>> host will be ignored, maybe you caught it when you created your
>>>>> own.
>>>>>
>>>>> I would try something like (for a 6 node host cluster):
>>>>>
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=sensors,ambari_
>>>>> master,ec2,monit
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=pcap_server,monit,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,
>>>>> enrichment,metron,ec2,zeppelin
>>>>> - include: tasks/create-hosts.yml host_count=2
>>>>> host_type=ambari_slave,search,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,web,ec2
>>>>>
>>>>> -D...
>>>>>
>>>>>
>>>>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be>
>>>>> wrote:
>>>>>
>>>>> Different error than before, but it still fails:
>>>>>
>>>>>>
>>>>>> TASK [ambari_gather_facts : set_fact]
>>>>>> ******************************
>>>>>> ************
>>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>>>>
>>>>>> TASK [ambari_gather_facts : debug] ******************************
>>>>>> ***************
>>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>>>>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url
>>>>>> =
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>>>>> kafka_broker_port
>>>>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>>>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>>>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>>>>> ompute.amazonaws.com'], kafka_broker_url =
>>>>>> ec2-34-210-18-244.us-west-2.co
>>>>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>>>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>>>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>>>>> 2.compute.amazonaws.com']"
>>>>>> }
>>>>>>
>>>>>> TASK [load_web_templates : Load ES Templates]
>>>>>> **********************************
>>>>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}",
>>>>>> "-u",
>>>>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d",
>>>>>> "{
>>>>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>>>>> \"command\":
>>>>>> \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>>>>> filters\":
>>>>>> [{\"service_name\": \"METRON\",\"component_name\":
>>>>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>>>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>>>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>>>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>>>>>> "failed":
>>>>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>>>>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>>>>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>>>>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>>>>>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202",
>>>>>> "stdout_lines":
>>>>>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>>>>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/
>>>>>> metron/requests/14%5C>
>>>>>>
>>>>>> ",",
>>>>>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>>>>>> \"Accepted\"",
>>>>>> " }", "}202"], "warnings": []}
>>>>>>
>>>>>> TASK [load_web_templates : Load Kibana Dashboard]
>>>>>> ******************************
>>>>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]:
>>>>>> FAILED!
>>>>>> =>
>>>>>> {"failed": true, "msg": "the field 'args' has an invalid value,
>>>>>> which
>>>>>> appears to include a variable that is undefined. The error was:
>>>>>> list
>>>>>> object
>>>>>> has no element 0\n\nThe error appears to have been in
>>>>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>>>>> tasks/main.yml':
>>>>>> line 26, column 3, but may\nbe elsewhere in the file depending on
>>>>>> the
>>>>>> exact
>>>>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name:
>>>>>> Load
>>>>>> Kibana Dashboard\n ^ here\n"}
>>>>>> to retry, use: --limit @/root/metron/metron-deploymen
>>>>>> t/amazon-ec2/playbook.retry
>>>>>>
>>>>>> PLAY RECAP ******************************
>>>>>> ******************************
>>>>>> *********
>>>>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>>>>>> changed=31
>>>>>> unreachable=0 failed=1
>>>>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>>>>>> changed=31
>>>>>> unreachable=0 failed=0
>>>>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> localhost : ok=16 changed=11 unreachable=0
>>>>>> failed=0
>>>>>>
>>>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>>>
>>>>>>
>>>>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>>>>
>>>>>> Maybe you could re-purpose the playbook.yml for the 10-node
>>>>>> install -
>>>>>>
>>>>>>> comment out some of the host creation lines and move the host
>>>>>>> types
>>>>>>> where
>>>>>>> you want them installed, e.g.
>>>>>>>
>>>>>>> From this
>>>>>>>
>>>>>>> tasks:
>>>>>>> - include: tasks/create-keypair.yml
>>>>>>> - include: tasks/create-vpc.yml
>>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>>> - include: tasks/create-hosts.yml host_count=4
>>>>>>> host_type=ambari_slave,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=pcap_server,monit,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>> - include: tasks/create-hosts.yml host_count=2
>>>>>>> host_type=ambari_slave,search,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,web,ec2
>>>>>>>
>>>>>>> to this (6-node)
>>>>>>>
>>>>>>> tasks:
>>>>>>> - include: tasks/create-keypair.yml
>>>>>>> - include: tasks/create-vpc.yml
>>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>>>>> - include: tasks/create-hosts.yml host_count=3
>>>>>>> host_type=ambari_slave,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>>>>>> ambari_slave
>>>>>>> ,pcap_server,monit,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>>
>>>>>>> Of course, this is just to demonstrate re-locating items, not
>>>>>>> necessarily
>>>>>>> how you would want to lay them out.
>>>>>>>
>>>>>>> Best,
>>>>>>> Mike Miklavcic
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets
>>>>>>> <la...@daemon.be>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>>>>>> default
>>>>>>>
>>>>>>> 10 node cluster install... So far no luck. Whatever I do, it
>>>>>>> always
>>>>>>>> ends
>>>>>>>> in
>>>>>>>> errors...
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>>>>
>>>>>>>> I'm gonna to venture a 'no' simply because I don't know what
>>>>>>>> that is.
>>>>>>>>
>>>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>>>>>>>> trying
>>>>>>>>> to
>>>>>>>>> do?
>>>>>>>>>
>>>>>>>>> -D...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets
>>>>>>>>> <la...@daemon.be>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Is AWS single_node_vm still supported?
Re: AWS single_node_vm
Posted by David Lyle <dl...@gmail.com>.
This one should do the trick for you.
Due to how smal_cluster.yml is structured (designed for 10+ nodes)- you'll
only get 2 data nodes, so you'll have to reduce replication to 2 if you
don't want to see under-replicated block warnings.
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ambari_slave,web,ec2
The reason the first one didn't work is because order matters. In the that
example, the host created by
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
was ending up in 2 separate hostgroups, which is not allowed. Relocating
that line solves the problem. I also relocated the web hosts (ES Master and
Kibana) since there were fewer hosts.
-D...
On Tue, May 30, 2017 at 12:49 PM, Laurens Vets <la...@daemon.be> wrote:
> I put in what you said previously :)
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,web,ec2
>
> On 2017-05-30 08:53, David Lyle wrote:
>
>> That's actually a different error than you posted before. That error is
>> Ambari telling you that the blueprint that Ansible submitted is invalid.
>> Likely a result of a required hostgroup not being defined. Can you (re?)
>> post what you've put in amazon-ec2/playbook.yml?
>>
>> -D...
>>
>>
>> On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <la...@daemon.be> wrote:
>>
>> Same problem as all the previous ones...
>>>
>>> TASK [ambari_config : Install python-requests]
>>> *********************************
>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"attempts":
>>> 1, "changed": false, "msg": "", "rc": 0, "results":
>>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is
>>> already
>>> installed"]}
>>>
>>> TASK [ambari_config : check if ambari-server is up on
>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed":
>>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
>>> "state": "started"}
>>>
>>> TASK [ambari_config : Deploy cluster with Ambari;
>>> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>>> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
>>> {"changed": false, "failed": true, "msg": "Ambari client exception
>>> occurred: Could not create cluster: request code 400,
>>> request message {\n \"status\" : 400,\n \"message\" : \"Topology
>>> validation failed: org.apache.ambari.server.topol
>>> ogy.InvalidTopologyException:
>>> The following hosts are mapped to multiple host groups: [
>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
>>> names are converted to lowercase, case differences do not matter in
>>> Ambari
>>> deployments.\"\n}"}
>>> to retry, use: --limit @/root/metron/metron-deploymen
>>> t/amazon-ec2/playbook.retry
>>>
>>> PLAY RECAP ************************************************************
>>> *********
>>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
>>> unreachable=0 failed=0
>>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
>>> unreachable=0 failed=1
>>> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
>>> unreachable=0 failed=0
>>> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> localhost : ok=18 changed=13 unreachable=0
>>> failed=0
>>>
>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>
>>>
>>> On 2017-05-27 04:45, David Lyle wrote:
>>>
>>> For some reason, kibana_hosts is undefined, so the curl command that
>>>> loads
>>>> the Kibana dashboard doesn't contain the URL for Kibana. If you look at
>>>> small_cluster.yml, you'll see that the Kibana should have been installed
>>>> on
>>>> the first host in the "web" hostgroup. Verify that you placed a host in
>>>> that hostgroup (it's the 5th line in Mike's example). If you did, check
>>>> that host on Ambari and verify that Kibana was installed, if not, you'll
>>>> want to dig through the install log to find out why.
>>>>
>>>> Also- now that I've looked at the example carefully, I noticed that the
>>>> Zeppelin hostgroup was duplicated. That's not a problem, because the 2nd
>>>> host will be ignored, maybe you caught it when you created your own.
>>>>
>>>> I would try something like (for a 6 node host cluster):
>>>>
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_
>>>> master,ec2,monit
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=pcap_server,monit,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,
>>>> enrichment,metron,ec2,zeppelin
>>>> - include: tasks/create-hosts.yml host_count=2
>>>> host_type=ambari_slave,search,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,web,ec2
>>>>
>>>> -D...
>>>>
>>>>
>>>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> Different error than before, but it still fails:
>>>>
>>>>>
>>>>> TASK [ambari_gather_facts : set_fact] ******************************
>>>>> ************
>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>>>
>>>>> TASK [ambari_gather_facts : debug] ******************************
>>>>> ***************
>>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>>>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>>>> kafka_broker_port
>>>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>>>> ompute.amazonaws.com'], kafka_broker_url =
>>>>> ec2-34-210-18-244.us-west-2.co
>>>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>>>> 2.compute.amazonaws.com']"
>>>>> }
>>>>>
>>>>> TASK [load_web_templates : Load ES Templates]
>>>>> **********************************
>>>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
>>>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
>>>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>>>> \"command\": \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>>>> filters\":
>>>>> [{\"service_name\": \"METRON\",\"component_name\":
>>>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>>>>> "failed":
>>>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>>>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>>>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>>>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>>>>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202",
>>>>> "stdout_lines":
>>>>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>>>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/
>>>>> metron/requests/14%5C>
>>>>>
>>>>> ",",
>>>>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>>>>> \"Accepted\"",
>>>>> " }", "}202"], "warnings": []}
>>>>>
>>>>> TASK [load_web_templates : Load Kibana Dashboard]
>>>>> ******************************
>>>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED!
>>>>> =>
>>>>> {"failed": true, "msg": "the field 'args' has an invalid value, which
>>>>> appears to include a variable that is undefined. The error was: list
>>>>> object
>>>>> has no element 0\n\nThe error appears to have been in
>>>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>>>> tasks/main.yml':
>>>>> line 26, column 3, but may\nbe elsewhere in the file depending on the
>>>>> exact
>>>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
>>>>> Kibana Dashboard\n ^ here\n"}
>>>>> to retry, use: --limit @/root/metron/metron-deploymen
>>>>> t/amazon-ec2/playbook.retry
>>>>>
>>>>> PLAY RECAP ******************************
>>>>> ******************************
>>>>> *********
>>>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>>>>> changed=31
>>>>> unreachable=0 failed=1
>>>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>>>>> changed=31
>>>>> unreachable=0 failed=0
>>>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>>>> unreachable=0 failed=0
>>>>> localhost : ok=16 changed=11 unreachable=0
>>>>> failed=0
>>>>>
>>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>>
>>>>>
>>>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>>>
>>>>> Maybe you could re-purpose the playbook.yml for the 10-node install -
>>>>>
>>>>>> comment out some of the host creation lines and move the host types
>>>>>> where
>>>>>> you want them installed, e.g.
>>>>>>
>>>>>> From this
>>>>>>
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>> - include: tasks/create-hosts.yml host_count=4
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>> - include: tasks/create-hosts.yml host_count=2
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,web,ec2
>>>>>>
>>>>>> to this (6-node)
>>>>>>
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>>>> - include: tasks/create-hosts.yml host_count=3
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>>>>> ambari_slave
>>>>>> ,pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>
>>>>>> Of course, this is just to demonstrate re-locating items, not
>>>>>> necessarily
>>>>>> how you would want to lay them out.
>>>>>>
>>>>>> Best,
>>>>>> Mike Miklavcic
>>>>>>
>>>>>>
>>>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be>
>>>>>> wrote:
>>>>>>
>>>>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>>>>> default
>>>>>>
>>>>>> 10 node cluster install... So far no luck. Whatever I do, it always
>>>>>>> ends
>>>>>>> in
>>>>>>> errors...
>>>>>>>
>>>>>>>
>>>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>>>
>>>>>>> I'm gonna to venture a 'no' simply because I don't know what that is.
>>>>>>>
>>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>>>>>>> trying
>>>>>>>> to
>>>>>>>> do?
>>>>>>>>
>>>>>>>> -D...
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Is AWS single_node_vm still supported?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: AWS single_node_vm
Posted by Laurens Vets <la...@daemon.be>.
I put in what you said previously :)
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2
On 2017-05-30 08:53, David Lyle wrote:
> That's actually a different error than you posted before. That error is
> Ambari telling you that the blueprint that Ansible submitted is
> invalid.
> Likely a result of a required hostgroup not being defined. Can you
> (re?)
> post what you've put in amazon-ec2/playbook.yml?
>
> -D...
>
>
> On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> Same problem as all the previous ones...
>>
>> TASK [ambari_config : Install python-requests]
>> *********************************
>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] =>
>> {"attempts":
>> 1, "changed": false, "msg": "", "rc": 0, "results":
>> ["python-requests-2.6.0-3.el6.noarch providing python-requests is
>> already
>> installed"]}
>>
>> TASK [ambari_config : check if ambari-server is up on
>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed":
>> false, "elapsed": 120, "path": null, "port": 8080, "search_regex":
>> null,
>> "state": "started"}
>>
>> TASK [ambari_config : Deploy cluster with Ambari;
>> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
>> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
>> {"changed": false, "failed": true, "msg": "Ambari client exception
>> occurred: Could not create cluster: request code 400,
>> request message {\n \"status\" : 400,\n \"message\" : \"Topology
>> validation failed:
>> org.apache.ambari.server.topology.InvalidTopologyException:
>> The following hosts are mapped to multiple host groups: [
>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
>> names are converted to lowercase, case differences do not matter in
>> Ambari
>> deployments.\"\n}"}
>> to retry, use: --limit @/root/metron/metron-deploymen
>> t/amazon-ec2/playbook.retry
>>
>> PLAY RECAP
>> ************************************************************
>> *********
>> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
>> unreachable=0 failed=0
>> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
>> unreachable=0 failed=1
>> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
>> unreachable=0 failed=0
>> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> localhost : ok=18 changed=13 unreachable=0
>> failed=0
>>
>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>
>>
>> On 2017-05-27 04:45, David Lyle wrote:
>>
>>> For some reason, kibana_hosts is undefined, so the curl command that
>>> loads
>>> the Kibana dashboard doesn't contain the URL for Kibana. If you look
>>> at
>>> small_cluster.yml, you'll see that the Kibana should have been
>>> installed
>>> on
>>> the first host in the "web" hostgroup. Verify that you placed a host
>>> in
>>> that hostgroup (it's the 5th line in Mike's example). If you did,
>>> check
>>> that host on Ambari and verify that Kibana was installed, if not,
>>> you'll
>>> want to dig through the install log to find out why.
>>>
>>> Also- now that I've looked at the example carefully, I noticed that
>>> the
>>> Zeppelin hostgroup was duplicated. That's not a problem, because the
>>> 2nd
>>> host will be ignored, maybe you caught it when you created your own.
>>>
>>> I would try something like (for a 6 node host cluster):
>>>
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_
>>> master,ec2,monit
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=pcap_server,monit,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,
>>> enrichment,metron,ec2,zeppelin
>>> - include: tasks/create-hosts.yml host_count=2
>>> host_type=ambari_slave,search,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,web,ec2
>>>
>>> -D...
>>>
>>>
>>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be>
>>> wrote:
>>>
>>> Different error than before, but it still fails:
>>>>
>>>> TASK [ambari_gather_facts : set_fact] ******************************
>>>> ************
>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>>
>>>> TASK [ambari_gather_facts : debug] ******************************
>>>> ***************
>>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>>> kafka_broker_port
>>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>>> ompute.amazonaws.com'], kafka_broker_url =
>>>> ec2-34-210-18-244.us-west-2.co
>>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>>> 2.compute.amazonaws.com']"
>>>> }
>>>>
>>>> TASK [load_web_templates : Load ES Templates]
>>>> **********************************
>>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
>>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d",
>>>> "{
>>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>>> \"command\":
>>>> \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>>> filters\":
>>>> [{\"service_name\": \"METRON\",\"component_name\":
>>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>>>> "failed":
>>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>>>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202",
>>>> "stdout_lines":
>>>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>>> ",",
>>>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>>>> \"Accepted\"",
>>>> " }", "}202"], "warnings": []}
>>>>
>>>> TASK [load_web_templates : Load Kibana Dashboard]
>>>> ******************************
>>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED!
>>>> =>
>>>> {"failed": true, "msg": "the field 'args' has an invalid value,
>>>> which
>>>> appears to include a variable that is undefined. The error was: list
>>>> object
>>>> has no element 0\n\nThe error appears to have been in
>>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>>> tasks/main.yml':
>>>> line 26, column 3, but may\nbe elsewhere in the file depending on
>>>> the
>>>> exact
>>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name:
>>>> Load
>>>> Kibana Dashboard\n ^ here\n"}
>>>> to retry, use: --limit @/root/metron/metron-deploymen
>>>> t/amazon-ec2/playbook.retry
>>>>
>>>> PLAY RECAP
>>>> ************************************************************
>>>> *********
>>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>>>> changed=31
>>>> unreachable=0 failed=1
>>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>>>> changed=31
>>>> unreachable=0 failed=0
>>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> localhost : ok=16 changed=11 unreachable=0
>>>> failed=0
>>>>
>>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>>
>>>>
>>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>>
>>>> Maybe you could re-purpose the playbook.yml for the 10-node install
>>>> -
>>>>> comment out some of the host creation lines and move the host types
>>>>> where
>>>>> you want them installed, e.g.
>>>>>
>>>>> From this
>>>>>
>>>>> tasks:
>>>>> - include: tasks/create-keypair.yml
>>>>> - include: tasks/create-vpc.yml
>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>> - include: tasks/create-hosts.yml host_count=4
>>>>> host_type=ambari_slave,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=pcap_server,monit,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>> - include: tasks/create-hosts.yml host_count=2
>>>>> host_type=ambari_slave,search,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,web,ec2
>>>>>
>>>>> to this (6-node)
>>>>>
>>>>> tasks:
>>>>> - include: tasks/create-keypair.yml
>>>>> - include: tasks/create-vpc.yml
>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>>> - include: tasks/create-hosts.yml host_count=3
>>>>> host_type=ambari_slave,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>>>> ambari_slave
>>>>> ,pcap_server,monit,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>
>>>>> Of course, this is just to demonstrate re-locating items, not
>>>>> necessarily
>>>>> how you would want to lay them out.
>>>>>
>>>>> Best,
>>>>> Mike Miklavcic
>>>>>
>>>>>
>>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be>
>>>>> wrote:
>>>>>
>>>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>>>> default
>>>>>
>>>>>> 10 node cluster install... So far no luck. Whatever I do, it
>>>>>> always
>>>>>> ends
>>>>>> in
>>>>>> errors...
>>>>>>
>>>>>>
>>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>>
>>>>>> I'm gonna to venture a 'no' simply because I don't know what that
>>>>>> is.
>>>>>>
>>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>>>>>> trying
>>>>>>> to
>>>>>>> do?
>>>>>>>
>>>>>>> -D...
>>>>>>>
>>>>>>>
>>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Is AWS single_node_vm still supported?
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
Re: AWS single_node_vm
Posted by David Lyle <dl...@gmail.com>.
That's actually a different error than you posted before. That error is
Ambari telling you that the blueprint that Ansible submitted is invalid.
Likely a result of a required hostgroup not being defined. Can you (re?)
post what you've put in amazon-ec2/playbook.yml?
-D...
On Mon, May 29, 2017 at 12:02 PM, Laurens Vets <la...@daemon.be> wrote:
> Same problem as all the previous ones...
>
> TASK [ambari_config : Install python-requests]
> *********************************
> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"attempts":
> 1, "changed": false, "msg": "", "rc": 0, "results":
> ["python-requests-2.6.0-3.el6.noarch providing python-requests is already
> installed"]}
>
> TASK [ambari_config : check if ambari-server is up on
> ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
> ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed":
> false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
> "state": "started"}
>
> TASK [ambari_config : Deploy cluster with Ambari;
> http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
> fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
> {"changed": false, "failed": true, "msg": "Ambari client exception
> occurred: Could not create cluster: request code 400,
> request message {\n \"status\" : 400,\n \"message\" : \"Topology
> validation failed: org.apache.ambari.server.topology.InvalidTopologyException:
> The following hosts are mapped to multiple host groups: [
> ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
> names are converted to lowercase, case differences do not matter in Ambari
> deployments.\"\n}"}
> to retry, use: --limit @/root/metron/metron-deploymen
> t/amazon-ec2/playbook.retry
>
> PLAY RECAP ************************************************************
> *********
> ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
> unreachable=0 failed=0
> ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
> unreachable=0 failed=1
> ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
> unreachable=0 failed=0
> ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> localhost : ok=18 changed=13 unreachable=0 failed=0
>
> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>
>
> On 2017-05-27 04:45, David Lyle wrote:
>
>> For some reason, kibana_hosts is undefined, so the curl command that loads
>> the Kibana dashboard doesn't contain the URL for Kibana. If you look at
>> small_cluster.yml, you'll see that the Kibana should have been installed
>> on
>> the first host in the "web" hostgroup. Verify that you placed a host in
>> that hostgroup (it's the 5th line in Mike's example). If you did, check
>> that host on Ambari and verify that Kibana was installed, if not, you'll
>> want to dig through the install log to find out why.
>>
>> Also- now that I've looked at the example carefully, I noticed that the
>> Zeppelin hostgroup was duplicated. That's not a problem, because the 2nd
>> host will be ignored, maybe you caught it when you created your own.
>>
>> I would try something like (for a 6 node host cluster):
>>
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_
>> master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,
>> enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,ec2
>>
>> -D...
>>
>>
>> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be> wrote:
>>
>> Different error than before, but it still fails:
>>>
>>> TASK [ambari_gather_facts : set_fact] ******************************
>>> ************
>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>>
>>> TASK [ambari_gather_facts : debug] ******************************
>>> ***************
>>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>>> kafka_broker_port
>>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>>> ompute.amazonaws.com'], kafka_broker_url =
>>> ec2-34-210-18-244.us-west-2.co
>>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>>> 2.compute.amazonaws.com']"
>>> }
>>>
>>> TASK [load_web_templates : Load ES Templates]
>>> **********************************
>>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
>>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
>>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>>> \"command\": \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_
>>> filters\":
>>> [{\"service_name\": \"METRON\",\"component_name\":
>>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106", "failed":
>>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202", "stdout_lines":
>>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\
>>> <http://west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14%5C>
>>> ",",
>>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>>> \"Accepted\"",
>>> " }", "}202"], "warnings": []}
>>>
>>> TASK [load_web_templates : Load Kibana Dashboard]
>>> ******************************
>>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED! =>
>>> {"failed": true, "msg": "the field 'args' has an invalid value, which
>>> appears to include a variable that is undefined. The error was: list
>>> object
>>> has no element 0\n\nThe error appears to have been in
>>> '/root/metron/metron-deployment/roles/load_web_templates/
>>> tasks/main.yml':
>>> line 26, column 3, but may\nbe elsewhere in the file depending on the
>>> exact
>>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
>>> Kibana Dashboard\n ^ here\n"}
>>> to retry, use: --limit @/root/metron/metron-deploymen
>>> t/amazon-ec2/playbook.retry
>>>
>>> PLAY RECAP ************************************************************
>>> *********
>>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80 changed=31
>>> unreachable=0 failed=1
>>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47 changed=31
>>> unreachable=0 failed=0
>>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
>>> unreachable=0 failed=0
>>> localhost : ok=16 changed=11 unreachable=0
>>> failed=0
>>>
>>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>>
>>>
>>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>>
>>> Maybe you could re-purpose the playbook.yml for the 10-node install -
>>>> comment out some of the host creation lines and move the host types
>>>> where
>>>> you want them installed, e.g.
>>>>
>>>> From this
>>>>
>>>> tasks:
>>>> - include: tasks/create-keypair.yml
>>>> - include: tasks/create-vpc.yml
>>>> - include: tasks/create-open-inbound-security-group.yml
>>>> - include: tasks/create-open-outbound-security-group.yml
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_master,ec2,monit
>>>> - include: tasks/create-hosts.yml host_count=4
>>>> host_type=ambari_slave,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=pcap_server,monit,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>> - include: tasks/create-hosts.yml host_count=2
>>>> host_type=ambari_slave,search,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,web,ec2
>>>>
>>>> to this (6-node)
>>>>
>>>> tasks:
>>>> - include: tasks/create-keypair.yml
>>>> - include: tasks/create-vpc.yml
>>>> - include: tasks/create-open-inbound-security-group.yml
>>>> - include: tasks/create-open-outbound-security-group.yml
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>>> - include: tasks/create-hosts.yml host_count=3
>>>> host_type=ambari_slave,ec2
>>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>>> ambari_slave
>>>> ,pcap_server,monit,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>
>>>> Of course, this is just to demonstrate re-locating items, not
>>>> necessarily
>>>> how you would want to lay them out.
>>>>
>>>> Best,
>>>> Mike Miklavcic
>>>>
>>>>
>>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>>> default
>>>>
>>>>> 10 node cluster install... So far no luck. Whatever I do, it always
>>>>> ends
>>>>> in
>>>>> errors...
>>>>>
>>>>>
>>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>>
>>>>> I'm gonna to venture a 'no' simply because I don't know what that is.
>>>>>
>>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you trying
>>>>>> to
>>>>>> do?
>>>>>>
>>>>>> -D...
>>>>>>
>>>>>>
>>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>>>>> wrote:
>>>>>>
>>>>>> Is AWS single_node_vm still supported?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
Re: AWS single_node_vm
Posted by Laurens Vets <la...@daemon.be>.
Same problem as all the previous ones...
TASK [ambari_config : Install python-requests]
*********************************
ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"attempts":
1, "changed": false, "msg": "", "rc": 0, "results":
["python-requests-2.6.0-3.el6.noarch providing python-requests is
already installed"]}
TASK [ambari_config : check if ambari-server is up on
ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
ok: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com] => {"changed":
false, "elapsed": 120, "path": null, "port": 8080, "search_regex": null,
"state": "started"}
TASK [ambari_config : Deploy cluster with Ambari;
http://ec2-34-209-23-182.us-west-2.compute.amazonaws.com:8080] ***
fatal: [ec2-34-209-23-182.us-west-2.compute.amazonaws.com]: FAILED! =>
{"changed": false, "failed": true, "msg": "Ambari client exception
occurred: Could not create cluster: request code 400,
request message {\n \"status\" : 400,\n \"message\" : \"Topology
validation failed:
org.apache.ambari.server.topology.InvalidTopologyException: The
following hosts are mapped to multiple host groups:
[ec2-34-208-74-17.us-west-2.compute.amazonaws.com]. Be aware that host
names are converted to lowercase, case differences do not matter in
Ambari deployments.\"\n}"}
to retry, use: --limit
@/root/metron/metron-deployment/amazon-ec2/playbook.retry
PLAY RECAP
*********************************************************************
ec2-34-208-74-17.us-west-2.compute.amazonaws.com : ok=47 changed=31
unreachable=0 failed=0
ec2-34-209-23-182.us-west-2.compute.amazonaws.com : ok=48 changed=28
unreachable=0 failed=1
ec2-35-163-138-4.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-35-163-150-6.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-35-167-207-80.us-west-2.compute.amazonaws.com : ok=17 changed=8
unreachable=0 failed=0
ec2-54-70-75-61.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
localhost : ok=18 changed=13 unreachable=0
failed=0
root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
On 2017-05-27 04:45, David Lyle wrote:
> For some reason, kibana_hosts is undefined, so the curl command that
> loads
> the Kibana dashboard doesn't contain the URL for Kibana. If you look at
> small_cluster.yml, you'll see that the Kibana should have been
> installed on
> the first host in the "web" hostgroup. Verify that you placed a host in
> that hostgroup (it's the 5th line in Mike's example). If you did, check
> that host on Ambari and verify that Kibana was installed, if not,
> you'll
> want to dig through the install log to find out why.
>
> Also- now that I've looked at the example carefully, I noticed that the
> Zeppelin hostgroup was duplicated. That's not a problem, because the
> 2nd
> host will be ignored, maybe you caught it when you created your own.
>
> I would try something like (for a 6 node host cluster):
>
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_
> master,ec2,monit
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,
> enrichment,metron,ec2,zeppelin
> - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,web,ec2
>
> -D...
>
>
> On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> Different error than before, but it still fails:
>>
>> TASK [ambari_gather_facts : set_fact] ******************************
>> ************
>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>>
>> TASK [ambari_gather_facts : debug] ******************************
>> ***************
>> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
>> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
>> kafka_broker_port
>> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
>> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
>> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
>> ompute.amazonaws.com'], kafka_broker_url =
>> ec2-34-210-18-244.us-west-2.co
>> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
>> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
>> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
>> 2.compute.amazonaws.com']"
>> }
>>
>> TASK [load_web_templates : Load ES Templates]
>> **********************************
>> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
>> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
>> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
>> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
>> \"command\":
>> \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_filters\":
>> [{\"service_name\": \"METRON\",\"component_name\":
>> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
>> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
>> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
>> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
>> "failed":
>> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
>> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
>> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
>> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
>> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202",
>> "stdout_lines":
>> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
>> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\",",
>> " \"Requests\" : {", " \"id\" : 14,", " \"status\" :
>> \"Accepted\"",
>> " }", "}202"], "warnings": []}
>>
>> TASK [load_web_templates : Load Kibana Dashboard]
>> ******************************
>> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED!
>> =>
>> {"failed": true, "msg": "the field 'args' has an invalid value, which
>> appears to include a variable that is undefined. The error was: list
>> object
>> has no element 0\n\nThe error appears to have been in
>> '/root/metron/metron-deployment/roles/load_web_templates/tasks/main.yml':
>> line 26, column 3, but may\nbe elsewhere in the file depending on the
>> exact
>> syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
>> Kibana Dashboard\n ^ here\n"}
>> to retry, use: --limit @/root/metron/metron-deploymen
>> t/amazon-ec2/playbook.retry
>>
>> PLAY RECAP
>> ************************************************************
>> *********
>> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80
>> changed=31
>> unreachable=0 failed=1
>> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47
>> changed=31
>> unreachable=0 failed=0
>> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
>> unreachable=0 failed=0
>> localhost : ok=16 changed=11 unreachable=0
>> failed=0
>>
>> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>>
>>
>> On 2017-05-26 11:59, Michael Miklavcic wrote:
>>
>>> Maybe you could re-purpose the playbook.yml for the 10-node install -
>>> comment out some of the host creation lines and move the host types
>>> where
>>> you want them installed, e.g.
>>>
>>> From this
>>>
>>> tasks:
>>> - include: tasks/create-keypair.yml
>>> - include: tasks/create-vpc.yml
>>> - include: tasks/create-open-inbound-security-group.yml
>>> - include: tasks/create-open-outbound-security-group.yml
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_master,ec2,monit
>>> - include: tasks/create-hosts.yml host_count=4
>>> host_type=ambari_slave,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=pcap_server,monit,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>> - include: tasks/create-hosts.yml host_count=2
>>> host_type=ambari_slave,search,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,web,ec2
>>>
>>> to this (6-node)
>>>
>>> tasks:
>>> - include: tasks/create-keypair.yml
>>> - include: tasks/create-vpc.yml
>>> - include: tasks/create-open-inbound-security-group.yml
>>> - include: tasks/create-open-outbound-security-group.yml
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>>> - include: tasks/create-hosts.yml host_count=3
>>> host_type=ambari_slave,ec2
>>> - include: tasks/create-hosts.yml host_count=1 host_type=
>>> ambari_slave
>>> ,pcap_server,monit,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>
>>> Of course, this is just to demonstrate re-locating items, not
>>> necessarily
>>> how you would want to lay them out.
>>>
>>> Best,
>>> Mike Miklavcic
>>>
>>>
>>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be>
>>> wrote:
>>>
>>> I'm trying to install Metron in AWS but with fewer hosts than the
>>> default
>>>> 10 node cluster install... So far no luck. Whatever I do, it always
>>>> ends
>>>> in
>>>> errors...
>>>>
>>>>
>>>> On 2017-05-26 10:16, David Lyle wrote:
>>>>
>>>> I'm gonna to venture a 'no' simply because I don't know what that
>>>> is.
>>>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>>>> trying
>>>>> to
>>>>> do?
>>>>>
>>>>> -D...
>>>>>
>>>>>
>>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>>>> wrote:
>>>>>
>>>>> Is AWS single_node_vm still supported?
>>>>>
>>>>>>
>>>>>>
>>>>>>
Re: AWS single_node_vm
Posted by David Lyle <dl...@gmail.com>.
For some reason, kibana_hosts is undefined, so the curl command that loads
the Kibana dashboard doesn't contain the URL for Kibana. If you look at
small_cluster.yml, you'll see that the Kibana should have been installed on
the first host in the "web" hostgroup. Verify that you placed a host in
that hostgroup (it's the 5th line in Mike's example). If you did, check
that host on Ambari and verify that Kibana was installed, if not, you'll
want to dig through the install log to find out why.
Also- now that I've looked at the example carefully, I noticed that the
Zeppelin hostgroup was duplicated. That's not a problem, because the 2nd
host will be ignored, maybe you caught it when you created your own.
I would try something like (for a 6 node host cluster):
- include: tasks/create-hosts.yml host_count=1 host_type=sensors,ambari_
master,ec2,monit
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1 host_type=ambari_slave,
enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2
-D...
On Fri, May 26, 2017 at 8:15 PM, Laurens Vets <la...@daemon.be> wrote:
> Different error than before, but it still fails:
>
> TASK [ambari_gather_facts : set_fact] ******************************
> ************
> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
> {"ansible_facts": {"kibana_hosts": []}, "changed": false}
>
> TASK [ambari_gather_facts : debug] ******************************
> ***************
> ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
> "msg": "zookeeper_port = 2181, zookeeper_hosts = [u'
> ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
> ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181, kafka_broker_port
> = 6667, kafka_broker_hosts = [u'ec2-34-210-18-244.us-west-2
> .compute.amazonaws.com', u'ec2-35-164-236-192.us-west-2
> .compute.amazonaws.com', u'ec2-52-41-30-101.us-west-2.c
> ompute.amazonaws.com'], kafka_broker_url = ec2-34-210-18-244.us-west-2.co
> mpute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compu
> te.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.
> amazonaws.com:6667, metron_hosts = [u'ec2-35-164-236-192.us-west-
> 2.compute.amazonaws.com']"
> }
>
> TASK [load_web_templates : Load ES Templates]
> **********************************
> changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
> {"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
> "admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
> \"RequestInfo\": { \"context\": \"Install ES Template from REST\",
> \"command\": \"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_filters\":
> [{\"service_name\": \"METRON\",\"component_name\":
> \"METRON_INDEXING\",\"hosts\" : \"ec2-35-164-236-192.us-west-2
> .compute.amazonaws.com\"}]}", "http://ec2-34-210-207-190.us-
> west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
> "delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106", "failed":
> false, "failed_when_result": false, "rc": 0, "start": "2017-05-26
> 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" : \"
> http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:
> 8080/api/v1/clusters/metron/requests/14\",\n \"Requests\" : {\n
> \"id\" : 14,\n \"status\" : \"Accepted\"\n }\n}202", "stdout_lines":
> ["{", " \"href\" : \"http://ec2-34-210-207-190.us
> -west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\",",
> " \"Requests\" : {", " \"id\" : 14,", " \"status\" : \"Accepted\"",
> " }", "}202"], "warnings": []}
>
> TASK [load_web_templates : Load Kibana Dashboard]
> ******************************
> fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED! =>
> {"failed": true, "msg": "the field 'args' has an invalid value, which
> appears to include a variable that is undefined. The error was: list object
> has no element 0\n\nThe error appears to have been in
> '/root/metron/metron-deployment/roles/load_web_templates/tasks/main.yml':
> line 26, column 3, but may\nbe elsewhere in the file depending on the exact
> syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Load
> Kibana Dashboard\n ^ here\n"}
> to retry, use: --limit @/root/metron/metron-deploymen
> t/amazon-ec2/playbook.retry
>
> PLAY RECAP ************************************************************
> *********
> ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80 changed=31
> unreachable=0 failed=1
> ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47 changed=31
> unreachable=0 failed=0
> ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
> unreachable=0 failed=0
> localhost : ok=16 changed=11 unreachable=0 failed=0
>
> root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
>
>
> On 2017-05-26 11:59, Michael Miklavcic wrote:
>
>> Maybe you could re-purpose the playbook.yml for the 10-node install -
>> comment out some of the host creation lines and move the host types where
>> you want them installed, e.g.
>>
>> From this
>>
>> tasks:
>> - include: tasks/create-keypair.yml
>> - include: tasks/create-vpc.yml
>> - include: tasks/create-open-inbound-security-group.yml
>> - include: tasks/create-open-outbound-security-group.yml
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> - include: tasks/create-hosts.yml host_count=4
>> host_type=ambari_slave,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,ec2
>>
>> to this (6-node)
>>
>> tasks:
>> - include: tasks/create-keypair.yml
>> - include: tasks/create-vpc.yml
>> - include: tasks/create-open-inbound-security-group.yml
>> - include: tasks/create-open-outbound-security-group.yml
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
>> - include: tasks/create-hosts.yml host_count=3
>> host_type=ambari_slave,ec2
>> - include: tasks/create-hosts.yml host_count=1 host_type= ambari_slave
>> ,pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>
>> Of course, this is just to demonstrate re-locating items, not necessarily
>> how you would want to lay them out.
>>
>> Best,
>> Mike Miklavcic
>>
>>
>> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be> wrote:
>>
>> I'm trying to install Metron in AWS but with fewer hosts than the default
>>> 10 node cluster install... So far no luck. Whatever I do, it always ends
>>> in
>>> errors...
>>>
>>>
>>> On 2017-05-26 10:16, David Lyle wrote:
>>>
>>> I'm gonna to venture a 'no' simply because I don't know what that is.
>>>> Afaik, single_node_vm.yml was never used with AWS. What are you trying
>>>> to
>>>> do?
>>>>
>>>> -D...
>>>>
>>>>
>>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> Is AWS single_node_vm still supported?
>>>>
>>>>>
>>>>>
>>>>>
Re: AWS single_node_vm
Posted by Laurens Vets <la...@daemon.be>.
Different error than before, but it still fails:
TASK [ambari_gather_facts : set_fact]
******************************************
ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
{"ansible_facts": {"kibana_hosts": []}, "changed": false}
TASK [ambari_gather_facts : debug]
*********************************************
ok: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] => {
"msg": "zookeeper_port = 2181, zookeeper_hosts =
[u'ec2-52-43-143-124.us-west-2.compute.amazonaws.com'], zookeeper_url =
ec2-52-43-143-124.us-west-2.compute.amazonaws.com:2181,
kafka_broker_port = 6667, kafka_broker_hosts =
[u'ec2-34-210-18-244.us-west-2.compute.amazonaws.com',
u'ec2-35-164-236-192.us-west-2.compute.amazonaws.com',
u'ec2-52-41-30-101.us-west-2.compute.amazonaws.com'], kafka_broker_url =
ec2-34-210-18-244.us-west-2.compute.amazonaws.com:6667,ec2-35-164-236-192.us-west-2.compute.amazonaws.com:6667,ec2-52-41-30-101.us-west-2.compute.amazonaws.com:6667,
metron_hosts = [u'ec2-35-164-236-192.us-west-2.compute.amazonaws.com']"
}
TASK [load_web_templates : Load ES Templates]
**********************************
changed: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com] =>
{"changed": true, "cmd": ["curl", "-s", "-w", "%{http_code}", "-u",
"admin:admin", "-H", "X-Requested-By: ambari", "-X", "POST", "-d", "{
\"RequestInfo\": { \"context\": \"Install ES Template from REST\",
\"command\":
\"ELASTICSEARCH_TEMPLATE_INSTALL\"},\"Requests/resource_filters\":
[{\"service_name\": \"METRON\",\"component_name\":
\"METRON_INDEXING\",\"hosts\" :
\"ec2-35-164-236-192.us-west-2.compute.amazonaws.com\"}]}",
"http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests"],
"delta": "0:00:00.049367", "end": "2017-05-26 21:33:34.767106",
"failed": false, "failed_when_result": false, "rc": 0, "start":
"2017-05-26 21:33:34.717739", "stderr": "", "stdout": "{\n \"href\" :
\"http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\",\n
\"Requests\" : {\n \"id\" : 14,\n \"status\" : \"Accepted\"\n
}\n}202", "stdout_lines": ["{", " \"href\" :
\"http://ec2-34-210-207-190.us-west-2.compute.amazonaws.com:8080/api/v1/clusters/metron/requests/14\",",
" \"Requests\" : {", " \"id\" : 14,", " \"status\" :
\"Accepted\"", " }", "}202"], "warnings": []}
TASK [load_web_templates : Load Kibana Dashboard]
******************************
fatal: [ec2-34-210-207-190.us-west-2.compute.amazonaws.com]: FAILED! =>
{"failed": true, "msg": "the field 'args' has an invalid value, which
appears to include a variable that is undefined. The error was: list
object has no element 0\n\nThe error appears to have been in
'/root/metron/metron-deployment/roles/load_web_templates/tasks/main.yml':
line 26, column 3, but may\nbe elsewhere in the file depending on the
exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name:
Load Kibana Dashboard\n ^ here\n"}
to retry, use: --limit
@/root/metron/metron-deployment/amazon-ec2/playbook.retry
PLAY RECAP
*********************************************************************
ec2-34-210-18-244.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-34-210-207-190.us-west-2.compute.amazonaws.com : ok=80 changed=31
unreachable=0 failed=1
ec2-34-211-19-168.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-35-164-236-192.us-west-2.compute.amazonaws.com : ok=47 changed=31
unreachable=0 failed=0
ec2-52-41-30-101.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
ec2-52-43-143-124.us-west-2.compute.amazonaws.com : ok=41 changed=27
unreachable=0 failed=0
localhost : ok=16 changed=11 unreachable=0
failed=0
root@ip-10-0-10-6:~/metron/metron-deployment/amazon-ec2#
On 2017-05-26 11:59, Michael Miklavcic wrote:
> Maybe you could re-purpose the playbook.yml for the 10-node install -
> comment out some of the host creation lines and move the host types
> where
> you want them installed, e.g.
>
> From this
>
> tasks:
> - include: tasks/create-keypair.yml
> - include: tasks/create-vpc.yml
> - include: tasks/create-open-inbound-security-group.yml
> - include: tasks/create-open-outbound-security-group.yml
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
> - include: tasks/create-hosts.yml host_count=4
> host_type=ambari_slave,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,web,ec2
>
> to this (6-node)
>
> tasks:
> - include: tasks/create-keypair.yml
> - include: tasks/create-vpc.yml
> - include: tasks/create-open-inbound-security-group.yml
> - include: tasks/create-open-outbound-security-group.yml
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
> - include: tasks/create-hosts.yml host_count=3
> host_type=ambari_slave,ec2
> - include: tasks/create-hosts.yml host_count=1 host_type=
> ambari_slave
> ,pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>
> Of course, this is just to demonstrate re-locating items, not
> necessarily
> how you would want to lay them out.
>
> Best,
> Mike Miklavcic
>
>
> On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> I'm trying to install Metron in AWS but with fewer hosts than the
>> default
>> 10 node cluster install... So far no luck. Whatever I do, it always
>> ends in
>> errors...
>>
>>
>> On 2017-05-26 10:16, David Lyle wrote:
>>
>>> I'm gonna to venture a 'no' simply because I don't know what that is.
>>> Afaik, single_node_vm.yml was never used with AWS. What are you
>>> trying to
>>> do?
>>>
>>> -D...
>>>
>>>
>>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
>>> wrote:
>>>
>>> Is AWS single_node_vm still supported?
>>>>
>>>>
Re: AWS single_node_vm
Posted by Michael Miklavcic <mi...@gmail.com>.
Maybe you could re-purpose the playbook.yml for the 10-node install -
comment out some of the host creation lines and move the host types where
you want them installed, e.g.
From this
tasks:
- include: tasks/create-keypair.yml
- include: tasks/create-vpc.yml
- include: tasks/create-open-inbound-security-group.yml
- include: tasks/create-open-outbound-security-group.yml
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
- include: tasks/create-hosts.yml host_count=4
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
- include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2
to this (6-node)
tasks:
- include: tasks/create-keypair.yml
- include: tasks/create-vpc.yml
- include: tasks/create-open-inbound-security-group.yml
- include: tasks/create-open-outbound-security-group.yml
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit,zeppelin,search,web
- include: tasks/create-hosts.yml host_count=3
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1 host_type= ambari_slave
,pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
Of course, this is just to demonstrate re-locating items, not necessarily
how you would want to lay them out.
Best,
Mike Miklavcic
On Fri, May 26, 2017 at 11:28 AM, Laurens Vets <la...@daemon.be> wrote:
> I'm trying to install Metron in AWS but with fewer hosts than the default
> 10 node cluster install... So far no luck. Whatever I do, it always ends in
> errors...
>
>
> On 2017-05-26 10:16, David Lyle wrote:
>
>> I'm gonna to venture a 'no' simply because I don't know what that is.
>> Afaik, single_node_vm.yml was never used with AWS. What are you trying to
>> do?
>>
>> -D...
>>
>>
>> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be> wrote:
>>
>> Is AWS single_node_vm still supported?
>>>
>>>
Re: AWS single_node_vm
Posted by Laurens Vets <la...@daemon.be>.
I'm trying to install Metron in AWS but with fewer hosts than the
default 10 node cluster install... So far no luck. Whatever I do, it
always ends in errors...
On 2017-05-26 10:16, David Lyle wrote:
> I'm gonna to venture a 'no' simply because I don't know what that is.
> Afaik, single_node_vm.yml was never used with AWS. What are you trying
> to
> do?
>
> -D...
>
>
> On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> Is AWS single_node_vm still supported?
>>
Re: AWS single_node_vm
Posted by David Lyle <dl...@gmail.com>.
I'm gonna to venture a 'no' simply because I don't know what that is.
Afaik, single_node_vm.yml was never used with AWS. What are you trying to
do?
-D...
On Thu, May 25, 2017 at 8:50 PM, Laurens Vets <la...@daemon.be> wrote:
> Is AWS single_node_vm still supported?
>