You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Laurens Vets <la...@daemon.be> on 2017/05/17 15:31:28 UTC
Trying to spin up Metron in EC2: Failed
Hello list,
I'm trying to spin up Metron in EC2 for testing purposes. I made the
following changes in playbook.yml so that I'm left with 5 EC2 instances
instead of 10:
<-- SNIP -->
tasks:
- include: tasks/create-keypair.yml
- include: tasks/create-vpc.yml
- include: tasks/create-open-inbound-security-group.yml
- include: tasks/create-open-outbound-security-group.yml
- include: tasks/create-hosts.yml host_count=1
host_type=sensors,ambari_master,ec2,monit
# - include: tasks/create-hosts.yml host_count=4
host_type=ambari_slave,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=pcap_server,monit,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,enrichment,metron,ec2,zeppelin
# - include: tasks/create-hosts.yml host_count=2
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,search,ec2
- include: tasks/create-hosts.yml host_count=1
host_type=ambari_slave,web,ec2
<-- END SNIP -->
I also had to change the line "ansible-playbook -vvv -i ec2.py
playbook.yml \
--skip-tags="quick_dev,sensor-stubs" \
--extra-vars="env=$ENV" \
$EXTRA_ARGS"
to "ansible-playbook -vvv -i ec2.py playbook.yml \
--skip-tags="quick_dev,sensor-stubs" \
--extra-vars="env=$ENV" \
$EXTRA_ARGS -c paramiko"
in run.sh otherwise ssh would time out.
However, Ansible errors out with:
2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config : Deploy
cluster with Ambari;
http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
/home/laurens/SAPSource/metron/metron-deployment/roles/ambari_config/tasks/main.yml:36
2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
[ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
{"changed": false, "failed": true, "invocation": {"module_args":
{"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
[{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
"SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
"HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
{"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
"ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
"configuration": [], "name": "master_1"}, {"cardinality": 1,
"components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
{"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
"SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
"HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": "MAPREDUCE2_CLIENT"},
{"name": "SPARK_CLIENT"}, {"name": "ZOOKEEPER_CLIENT"}, {"name":
"HBASE_CLIENT"}, {"name": "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}],
"configuration": [], "name": "master_2"}, {"cardinality": 1,
"components": [{"name": "METRON_INDEXING"}, {"name":
"METRON_ENRICHMENT_MASTER"}, {"name": "METRON_PARSERS"}, {"name":
"METRON_REST"}, {"name": "DATANODE"}, {"name": "NODEMANAGER"}, {"name":
"SUPERVISOR"}, {"name": "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"},
{"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
"MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
"ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
"name": "metron"}, {"cardinality": 1, "components": [{"name":
"KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name":
"web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
{"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
"KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
"HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name": "MAPREDUCE2_CLIENT"},
{"name": "SPARK_CLIENT"}, {"name": "ZOOKEEPER_CLIENT"}, {"name":
"HBASE_CLIENT"}], "configuration": [], "name": "slaves"}],
"required_configurations": [{"metron-env": {"es_hosts":
"ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-208-94-45.us-west-2.compute.amazonaws.com",
"metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root",
"metron_jdbc_platform": "h2", "metron_jdbc_url":
"jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
"storm_rest_addr":
"ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
"zeppelin_server_url":
"ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
{"kibana_default_application": "dashboard/Metron-Dashboard",
"kibana_es_url":
"http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
"kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
"/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP",
"stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
"present", "configurations": [{"zoo.cfg": {"dataDir":
"/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
"namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
{"dfs.datanode.data.dir":
"/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
"dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
"dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
"dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
{"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
"/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
"mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
{"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
"yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
"yarn.nodemanager.resource.memory-mb": 4096,
"yarn.timeline-service.leveldb-state-store.path":
"/data1/hadoop/yarn/timeline",
"yarn.timeline-service.leveldb-timeline-store.path":
"/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
"/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
{"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
{"parsers": "bro,snort,yaf"}}, {"elastic-site":
{"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1,
"index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
"zen_discovery_ping_unicast_hosts": "[
ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
"ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
"admin", "port": 8080, "username": "admin", "wait_for_complete": true},
"module_name": "ambari_cluster_state"}, "msg": "Ambari client exception
occurred: Could not create cluster: request code 400,
request message {\n \"status\" : 400,\n \"message\" : \"Invalid
Cluster Creation Template:
org.apache.ambari.server.topology.InvalidTopologyTemplateException: Host
group 'slaves' must contain at least one 'hosts/fqdn' or a 'host_count'
value\"\n}"}
PLAY RECAP
*********************************************************************
ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
unreachable=0 failed=0
ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8
unreachable=0 failed=0
ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31
unreachable=0 failed=0
ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27
unreachable=0 failed=0
ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28
unreachable=0 failed=1
localhost : ok=29 changed=12 unreachable=0
failed=0
Any idea what might be going on?
(Don't worry about the URLs, the instances have been terminated)
Re: Trying to spin up Metron in EC2: Failed
Posted by Laurens Vets <la...@daemon.be>.
Thanks, I'll try your suggestion of deploying from AWS itself.
On 2017-05-19 10:52, Nick Allen wrote:
> I am not sure what the issue is. Nothing is jumping out at me.
>
> Weird issues can arise when the machine that you are running the
> Ansible
> deployment from has intermittent connectivity issues to the AWS data
> center.
>
> Is Amazon's us-west-2 data center the one closest to you? You might
> try a
> different/closer data center. You can alter the data center here [1].
>
> [1]
> https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L27-L39
>
> Other than that, you could try standing up a small node in AWS and then
> use
> that to run the Metron deployment from. That always works much more
> smoothly.
>
>
>
>
> On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> Also, it's only 1 out of 10 hosts that failing it seems.
>>
>>
>> On 2017-05-17 13:25, Laurens Vets wrote:
>>
>>> When you say Metron host, do you mean the host I'm pushing from to
>>> ec2
>>> or the master ambari server in ec2?
>>>
>>> On 2017-05-17 11:51, Ryan Merriman wrote:
>>>
>>>> That happens when you don't have the zookeeper url configured
>>>> correctly.
>>>> Can you check the contents of the /etc/sysconfig/metron file on the
>>>> Metron
>>>> host?
>>>>
>>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> For testing purposes, I decided to spin up the default Metron AWS
>>>> config.
>>>>> This resulted in a hang from ansible here:
>>>>>
>>>>> TASK [librdkafka : include] ******************************
>>>>> **********************
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/main.yml:18
>>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml for
>>>>> ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>>
>>>>> TASK [librdkafka : Install prerequisites]
>>>>> ******************************
>>>>> ********
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml:18
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>>>> CONNECTION
>>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>>>> FutureWarning: CTR mode needs counter parameter, not IV
>>>>> self._cipher = factory.new(key, *args, **kwargs)
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask
>>>>> 22 &&
>>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" )
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>>>>> /tmp/tmpwnH61y
>>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh
>>>>> -c
>>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>>>> /usr/bin/python
>>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum;
>>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385
>>>>> 3889508/"
>>>>> > /dev/null 2>&1'"'"''
>>>>>
>>>>> Looking in the machine logs, I see the following for Kafka and
>>>>> Metron
>>>>> REST:
>>>>>
>>>>> Kafka:
>>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>>>> advertised.host.name = null
>>>>> metric.reporters = []
>>>>> quota.producer.default = 9223372036854775807
>>>>> offsets.topic.num.partitions = 50
>>>>> log.flush.interval.messages = 9223372036854775807
>>>>> auto.create.topics.enable = true
>>>>> controller.socket.timeout.ms = 30000
>>>>> log.flush.interval.ms = null
>>>>> principal.builder.class = class
>>>>> org.apache.kafka.common.securi
>>>>> ty.auth.DefaultPrincipalBuilder
>>>>> replica.socket.receive.buffer.bytes = 65536
>>>>> min.insync.replicas = 1
>>>>> replica.fetch.wait.max.ms = 500
>>>>> num.recovery.threads.per.data.dir = 1
>>>>> ssl.keystore.type = JKS
>>>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>>>> default.replication.factor = 1
>>>>> ssl.truststore.password = null
>>>>> log.preallocate = false
>>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>>>> fetch.purgatory.purge.interval.requests = 10000
>>>>> ssl.endpoint.identification.algorithm = null
>>>>> replica.socket.timeout.ms = 30000
>>>>> message.max.bytes = 1000000
>>>>> num.io.threads = 8
>>>>> offsets.commit.required.acks = -1
>>>>> log.flush.offset.checkpoint.interval.ms = 60000
>>>>> delete.topic.enable = false
>>>>> quota.window.size.seconds = 1
>>>>> ssl.truststore.type = JKS
>>>>> offsets.commit.timeout.ms = 5000
>>>>> quota.window.num = 11
>>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>>>> ompute.amazonaws.com:2181
>>>>> authorizer.class.name =
>>>>> num.replica.fetchers = 1
>>>>> log.retention.ms = null
>>>>> log.roll.jitter.hours = 0
>>>>> log.cleaner.enable = true
>>>>> offsets.load.buffer.size = 5242880
>>>>> log.cleaner.delete.retention.ms = 86400000
>>>>> ssl.client.auth = none
>>>>> controlled.shutdown.max.retries = 3
>>>>> queued.max.requests = 500
>>>>> offsets.topic.replication.factor = 3
>>>>> log.cleaner.threads = 1
>>>>> sasl.kerberos.service.name = null
>>>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>>>> socket.request.max.bytes = 104857600
>>>>> ssl.trustmanager.algorithm = PKIX
>>>>> zookeeper.session.timeout.ms = 30000
>>>>> log.retention.bytes = -1
>>>>> log.message.timestamp.type = CreateTime
>>>>> sasl.kerberos.min.time.before.relogin = 60000
>>>>> zookeeper.set.acl = false
>>>>> connections.max.idle.ms = 600000
>>>>> offsets.retention.minutes = 86400000
>>>>> replica.fetch.backoff.ms = 1000
>>>>> inter.broker.protocol.version = 0.10.0-IV1
>>>>> log.retention.hours = 168
>>>>> num.partitions = 1
>>>>> broker.id.generation.enable = true
>>>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>>>> us-west-2.compute.amazonaws.com:6667
>>>>> ssl.provider = null
>>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>>>> log.roll.ms = null
>>>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>>>> ssl.cipher.suites = null
>>>>> log.index.size.max.bytes = 10485760
>>>>> ssl.keymanager.algorithm = SunX509
>>>>> security.inter.broker.protocol = PLAINTEXT
>>>>> replica.fetch.max.bytes = 1048576
>>>>> advertised.port = null
>>>>> log.cleaner.dedupe.buffer.size = 134217728
>>>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>>>> log.cleaner.io.buffer.size = 524288
>>>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>>>> zookeeper.connection.timeout.ms = 25000
>>>>> controlled.shutdown.retry.backoff.ms = 5000
>>>>> log.roll.hours = 168
>>>>> log.cleanup.policy = delete
>>>>> host.name =
>>>>> log.roll.jitter.ms = null
>>>>> max.connections.per.ip = 2147483647
>>>>> offsets.topic.segment.bytes = 104857600
>>>>> background.threads = 10
>>>>> quota.consumer.default = 9223372036854775807
>>>>> request.timeout.ms = 30000
>>>>> log.message.format.version = 0.10.0-IV1
>>>>> log.index.interval.bytes = 4096
>>>>> log.dir = /tmp/kafka-logs
>>>>> log.segment.bytes = 1073741824
>>>>> log.cleaner.backoff.ms = 15000
>>>>> offset.metadata.max.bytes = 4096
>>>>> ssl.truststore.location = null
>>>>> group.max.session.timeout.ms = 300000
>>>>> ssl.keystore.password = null
>>>>> zookeeper.sync.time.ms = 2000
>>>>> port = 6667
>>>>> log.retention.minutes = null
>>>>> log.segment.delete.delay.ms = 60000
>>>>> log.dirs = /data2/kafka-log
>>>>> controlled.shutdown.enable = true
>>>>> compression.type = producer
>>>>> max.connections.per.ip.overrides =
>>>>> log.message.timestamp.difference.max.ms =
>>>>> 9223372036854775807
>>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>>>> log.cleaner.io.max.bytes.per.second =
>>>>> 1.7976931348623157E308
>>>>> auto.leader.rebalance.enable = true
>>>>> leader.imbalance.check.interval.seconds = 300
>>>>> log.cleaner.min.cleanable.ratio = 0.5
>>>>> replica.lag.time.max.ms = 10000
>>>>> num.network.threads = 3
>>>>> ssl.key.password = null
>>>>> reserved.broker.max.id = 1000
>>>>> metrics.num.samples = 2
>>>>> socket.send.buffer.bytes = 102400
>>>>> ssl.protocol = TLS
>>>>> socket.receive.buffer.bytes = 102400
>>>>> ssl.keystore.location = null
>>>>> replica.fetch.min.bytes = 1
>>>>> broker.rack = null
>>>>> unclean.leader.election.enable = true
>>>>> sasl.enabled.mechanisms = [GSSAPI]
>>>>> group.min.session.timeout.ms = 6000
>>>>> log.cleaner.io.buffer.load.factor = 0.9
>>>>> offsets.retention.check.interval.ms = 600000
>>>>> producer.purgatory.purge.interval.requests = 10000
>>>>> metrics.sample.window.ms = 30000
>>>>> broker.id = -1
>>>>> offsets.topic.compression.codec = 0
>>>>> log.retention.check.interval.ms = 300000
>>>>> advertised.listeners = null
>>>>> leader.imbalance.per.broker.percentage = 10
>>>>> (kafka.server.KafkaConfig)
>>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer
>>>>> startup.
>>>>> Prepare to shutdown (kafka.server.KafkaServer)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>> to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at
>>>>> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at
>>>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,740] INFO shutting down
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during
>>>>> KafkaServerStartable
>>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>> to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at
>>>>> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at
>>>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,764] INFO shutting down
>>>>> (kafka.server.KafkaServer)
>>>>>
>>>>> Manually starting Kafka via Ambari fixes this... However, Metron
>>>>> REST
>>>>> still doesn't work:
>>>>>
>>>>> . ____ _ __ _ _
>>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>>>> =========|_|==============|___/=/_/_/_/
>>>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>>>
>>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri
>>>>> ngApplication]
>>>>> - Application startup failed
>>>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>>>> Error
>>>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic
>>>>> eImpl.class]:
>>>>> Unsatisfied dependency expressed through constructor parameter 0;
>>>>> nested
>>>>> exception is org.springframework.beans.fact
>>>>> ory.UnsatisfiedDependencyException:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:749)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.autowireConstructor(ConstructorResolver.java:189)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.autowireConstructor(AbstractAu
>>>>> towireCapableBeanFactory.java:1148)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1051)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>>>>> at
>>>>> org.springframework.context.support.AbstractApplicationConte
>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
>>>>> at
>>>>> org.springframework.context.support.AbstractApplicationConte
>>>>> xt.refresh(AbstractApplicationContext.java:541)
>>>>> at
>>>>> org.springframework.boot.context.embedded.EmbeddedWebApplica
>>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.refresh(SpringApp
>>>>> lication.java:761)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.refreshContext(Sp
>>>>> ringApplication.java:371)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:315)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:1186)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:1175)
>>>>> at
>>>>> org.apache.metron.rest.MetronRestApplication.main(MetronRest
>>>>> Application.java:27)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.springframework.boot.loader.MainMethodRunner.run(MainMet
>>>>> hodRunner.java:48)
>>>>> at
>>>>> org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:87)
>>>>> at
>>>>> org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:50)
>>>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>>>> PropertiesLauncher.java:525)
>>>>> Caused by:
>>>>> org.springframework.beans.factory.UnsatisfiedDependencyExcep
>>>>> tion:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:569)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.InjectionMetada
>>>>> ta.inject(InjectionMetadata.java:88)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>>>> otationBeanPostProcessor.java:349)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:1219)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:543)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at
>>>>> org.springframework.beans.factory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:741)
>>>>> ... 27 more
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>>> Error
>>>>> creating bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at
>>>>> org.springframework.beans.factory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:566)
>>>>> ... 50 more
>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>> Failed
>>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>>>> ... 62 more
>>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable
>>>>> to
>>>>> connect to zookeeper server within timeout: 10000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
>>>>> eperConfig.java:52)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(
>>>>> <generated>)
>>>>> at
>>>>> org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
>>>>> odProxy.java:228)
>>>>> at
>>>>> org.springframework.context.annotation.ConfigurationClassEnh
>>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha
>>>>> ncer.java:356)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>>>> ... 63 more
>>>>>
>>>>> Any ideas on what might be going or why the timeout triggers?
>>>>>
>>>>>
>>>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>>>
>>>>> Hello list,
>>>>>>
>>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made
>>>>>> the
>>>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>>>> instances instead of 10:
>>>>>>
>>>>>> <-- SNIP -->
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>> # - include: tasks/create-hosts.yml host_count=4
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>> # - include: tasks/create-hosts.yml host_count=2
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,web,ec2
>>>>>> <-- END SNIP -->
>>>>>>
>>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>>>> playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS"
>>>>>>
>>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS -c paramiko"
>>>>>> in run.sh otherwise ssh would time out.
>>>>>>
>>>>>> However, Ansible errors out with:
>>>>>>
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>>>> Deploy cluster with Ambari;
>>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> ambari_config/tasks/main.yml:36
>>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name":
>>>>>> "YARN_CLIENT"},
>>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>> "ES_SLAVE"}],
>>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration":
>>>>>> [],
>>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration":
>>>>>> [],
>>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [],
>>>>>> "name":
>>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration":
>>>>>> [],
>>>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>>>> {"es_hosts":
>>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password":
>>>>>> "root",
>>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>>>> "storm_rest_addr":
>>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>>>> "zeppelin_server_url":
>>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}},
>>>>>> {"kibana-env":
>>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>>>> "kibana_es_url":
>>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name":
>>>>>> "HDP",
>>>>>> "stack_version": "2.5"}, "cluster_name": "metron",
>>>>>> "cluster_state":
>>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize":
>>>>>> 1024,
>>>>>> "namenode_heapsize": 2048}}, {"hbase-env":
>>>>>> {"hbase_master_heapsize":
>>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>>>> {"dfs.datanode.data.dir":
>>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>>>> {"mapred-site":
>>>>>> {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>>>> "/data1/hadoop/yarn/timeline",
>>>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site":
>>>>>> {"storm.local.dir":
>>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701,
>>>>>> 6702,
>>>>>> 6703]", "topology.classpath":
>>>>>> "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>>>> {"gateway_recover_after_data_nodes": 1,
>>>>>> "index_number_of_replicas": 1,
>>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_
>>>>>> ]",
>>>>>> "zen_discovery_ping_unicast_hosts": "[
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari
>>>>>> client
>>>>>> exception occurred: Could not create cluster: request code 400,
>>>>>> request message {\n \"status\" : 400,\n \"message\"
>>>>>> :
>>>>>> \"Invalid Cluster Creation Template:
>>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>>>> 'host_count' value\"\n}"}
>>>>>>
>>>>>> PLAY RECAP ******************************
>>>>>> ******************************
>>>>>> *********
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>>>>> changed=8
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>>>>> changed=31
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>>>>> changed=28
>>>>>> unreachable=0 failed=1
>>>>>> localhost : ok=29 changed=12 unreachable=0
>>>>>> failed=0
>>>>>>
>>>>>> Any idea what might be going on?
>>>>>>
>>>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>>>
>>>>>>
>>>>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Nick Allen <ni...@nickallen.org>.
Good. Glad to hear that.
On Tue, May 23, 2017 at 6:50 PM, Laurens Vets <la...@daemon.be> wrote:
> Hmmm, deploying Metron in AWS from another EC2 instance seems to work
> fine...
>
> I have a fully deployed 10 node Metron install now.
>
> On 2017-05-19 10:52, Nick Allen wrote:
>
>> I am not sure what the issue is. Nothing is jumping out at me.
>>
>> Weird issues can arise when the machine that you are running the Ansible
>> deployment from has intermittent connectivity issues to the AWS data
>> center.
>>
>> Is Amazon's us-west-2 data center the one closest to you? You might try a
>> different/closer data center. You can alter the data center here [1].
>>
>> [1]
>> https://github.com/apache/metron/blob/master/metron-deployme
>> nt/amazon-ec2/conf/defaults.yml#L27-L39
>>
>> Other than that, you could try standing up a small node in AWS and then
>> use
>> that to run the Metron deployment from. That always works much more
>> smoothly.
>>
>>
>>
>>
>> On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <la...@daemon.be> wrote:
>>
>> Also, it's only 1 out of 10 hosts that failing it seems.
>>>
>>>
>>> On 2017-05-17 13:25, Laurens Vets wrote:
>>>
>>> When you say Metron host, do you mean the host I'm pushing from to ec2
>>>> or the master ambari server in ec2?
>>>>
>>>> On 2017-05-17 11:51, Ryan Merriman wrote:
>>>>
>>>> That happens when you don't have the zookeeper url configured correctly.
>>>>> Can you check the contents of the /etc/sysconfig/metron file on the
>>>>> Metron
>>>>> host?
>>>>>
>>>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
>>>>> wrote:
>>>>>
>>>>> For testing purposes, I decided to spin up the default Metron AWS
>>>>> config.
>>>>>
>>>>>> This resulted in a hang from ansible here:
>>>>>>
>>>>>> TASK [librdkafka : include] ******************************
>>>>>> **********************
>>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> librdkafka/tasks/main.yml:18
>>>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
>>>>>> ompute.amazonaws.com
>>>>>>
>>>>>> TASK [librdkafka : Install prerequisites]
>>>>>> ******************************
>>>>>> ********
>>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> librdkafka/tasks/dependencies.yml:18
>>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>>>>> CONNECTION
>>>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>>>>> ompute.amazonaws.com
>>>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>>>>> FutureWarning: CTR mode needs counter parameter, not IV
>>>>>> self._cipher = factory.new(key, *args, **kwargs)
>>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22
>>>>>> &&
>>>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>>> -1495041091.74-92163853889508
>>>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>>> -1495041091.74-92163853889508
>>>>>> )" )
>>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>>>>>> /tmp/tmpwnH61y
>>>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>>> 889508/yum
>>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
>>>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>>>>> /usr/bin/python
>>>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>>> 889508/yum;
>>>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385
>>>>>> 3889508/"
>>>>>> > /dev/null 2>&1'"'"''
>>>>>>
>>>>>> Looking in the machine logs, I see the following for Kafka and Metron
>>>>>> REST:
>>>>>>
>>>>>> Kafka:
>>>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>>>>> advertised.host.name = null
>>>>>> metric.reporters = []
>>>>>> quota.producer.default = 9223372036854775807
>>>>>> offsets.topic.num.partitions = 50
>>>>>> log.flush.interval.messages = 9223372036854775807
>>>>>> auto.create.topics.enable = true
>>>>>> controller.socket.timeout.ms = 30000
>>>>>> log.flush.interval.ms = null
>>>>>> principal.builder.class = class org.apache.kafka.common.securi
>>>>>> ty.auth.DefaultPrincipalBuilder
>>>>>> replica.socket.receive.buffer.bytes = 65536
>>>>>> min.insync.replicas = 1
>>>>>> replica.fetch.wait.max.ms = 500
>>>>>> num.recovery.threads.per.data.dir = 1
>>>>>> ssl.keystore.type = JKS
>>>>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>>>>> default.replication.factor = 1
>>>>>> ssl.truststore.password = null
>>>>>> log.preallocate = false
>>>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>>>>> fetch.purgatory.purge.interval.requests = 10000
>>>>>> ssl.endpoint.identification.algorithm = null
>>>>>> replica.socket.timeout.ms = 30000
>>>>>> message.max.bytes = 1000000
>>>>>> num.io.threads = 8
>>>>>> offsets.commit.required.acks = -1
>>>>>> log.flush.offset.checkpoint.interval.ms = 60000
>>>>>> delete.topic.enable = false
>>>>>> quota.window.size.seconds = 1
>>>>>> ssl.truststore.type = JKS
>>>>>> offsets.commit.timeout.ms = 5000
>>>>>> quota.window.num = 11
>>>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>>>>> ompute.amazonaws.com:2181
>>>>>> authorizer.class.name =
>>>>>> num.replica.fetchers = 1
>>>>>> log.retention.ms = null
>>>>>> log.roll.jitter.hours = 0
>>>>>> log.cleaner.enable = true
>>>>>> offsets.load.buffer.size = 5242880
>>>>>> log.cleaner.delete.retention.ms = 86400000
>>>>>> ssl.client.auth = none
>>>>>> controlled.shutdown.max.retries = 3
>>>>>> queued.max.requests = 500
>>>>>> offsets.topic.replication.factor = 3
>>>>>> log.cleaner.threads = 1
>>>>>> sasl.kerberos.service.name = null
>>>>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>>>>> socket.request.max.bytes = 104857600
>>>>>> ssl.trustmanager.algorithm = PKIX
>>>>>> zookeeper.session.timeout.ms = 30000
>>>>>> log.retention.bytes = -1
>>>>>> log.message.timestamp.type = CreateTime
>>>>>> sasl.kerberos.min.time.before.relogin = 60000
>>>>>> zookeeper.set.acl = false
>>>>>> connections.max.idle.ms = 600000
>>>>>> offsets.retention.minutes = 86400000
>>>>>> replica.fetch.backoff.ms = 1000
>>>>>> inter.broker.protocol.version = 0.10.0-IV1
>>>>>> log.retention.hours = 168
>>>>>> num.partitions = 1
>>>>>> broker.id.generation.enable = true
>>>>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>>>>> us-west-2.compute.amazonaws.com:6667
>>>>>> ssl.provider = null
>>>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>>>>> log.roll.ms = null
>>>>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>>>>> ssl.cipher.suites = null
>>>>>> log.index.size.max.bytes = 10485760
>>>>>> ssl.keymanager.algorithm = SunX509
>>>>>> security.inter.broker.protocol = PLAINTEXT
>>>>>> replica.fetch.max.bytes = 1048576
>>>>>> advertised.port = null
>>>>>> log.cleaner.dedupe.buffer.size = 134217728
>>>>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>>>>> log.cleaner.io.buffer.size = 524288
>>>>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>>>>> zookeeper.connection.timeout.ms = 25000
>>>>>> controlled.shutdown.retry.backoff.ms = 5000
>>>>>> log.roll.hours = 168
>>>>>> log.cleanup.policy = delete
>>>>>> host.name =
>>>>>> log.roll.jitter.ms = null
>>>>>> max.connections.per.ip = 2147483647
>>>>>> offsets.topic.segment.bytes = 104857600
>>>>>> background.threads = 10
>>>>>> quota.consumer.default = 9223372036854775807
>>>>>> request.timeout.ms = 30000
>>>>>> log.message.format.version = 0.10.0-IV1
>>>>>> log.index.interval.bytes = 4096
>>>>>> log.dir = /tmp/kafka-logs
>>>>>> log.segment.bytes = 1073741824
>>>>>> log.cleaner.backoff.ms = 15000
>>>>>> offset.metadata.max.bytes = 4096
>>>>>> ssl.truststore.location = null
>>>>>> group.max.session.timeout.ms = 300000
>>>>>> ssl.keystore.password = null
>>>>>> zookeeper.sync.time.ms = 2000
>>>>>> port = 6667
>>>>>> log.retention.minutes = null
>>>>>> log.segment.delete.delay.ms = 60000
>>>>>> log.dirs = /data2/kafka-log
>>>>>> controlled.shutdown.enable = true
>>>>>> compression.type = producer
>>>>>> max.connections.per.ip.overrides =
>>>>>> log.message.timestamp.difference.max.ms = 9223372036854775807
>>>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>>>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>>>>>> auto.leader.rebalance.enable = true
>>>>>> leader.imbalance.check.interval.seconds = 300
>>>>>> log.cleaner.min.cleanable.ratio = 0.5
>>>>>> replica.lag.time.max.ms = 10000
>>>>>> num.network.threads = 3
>>>>>> ssl.key.password = null
>>>>>> reserved.broker.max.id = 1000
>>>>>> metrics.num.samples = 2
>>>>>> socket.send.buffer.bytes = 102400
>>>>>> ssl.protocol = TLS
>>>>>> socket.receive.buffer.bytes = 102400
>>>>>> ssl.keystore.location = null
>>>>>> replica.fetch.min.bytes = 1
>>>>>> broker.rack = null
>>>>>> unclean.leader.election.enable = true
>>>>>> sasl.enabled.mechanisms = [GSSAPI]
>>>>>> group.min.session.timeout.ms = 6000
>>>>>> log.cleaner.io.buffer.load.factor = 0.9
>>>>>> offsets.retention.check.interval.ms = 600000
>>>>>> producer.purgatory.purge.interval.requests = 10000
>>>>>> metrics.sample.window.ms = 30000
>>>>>> broker.id = -1
>>>>>> offsets.topic.compression.codec = 0
>>>>>> log.retention.check.interval.ms = 300000
>>>>>> advertised.listeners = null
>>>>>> leader.imbalance.per.broker.percentage = 10
>>>>>> (kafka.server.KafkaConfig)
>>>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>>>>> (kafka.server.KafkaServer)
>>>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer
>>>>>> startup.
>>>>>> Prepare to shutdown (kafka.server.KafkaServer)
>>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>>> to
>>>>>> zookeeper server within timeout: 25000
>>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>>> scala:75)
>>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>>> at kafka.server.KafkaServerStarta
>>>>>> ble.startup(KafkaServerStartab
>>>>>> le.scala:37)
>>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>>> [2017-05-17 17:03:40,740] INFO shutting down
>>>>>> (kafka.server.KafkaServer)
>>>>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>>>>> (kafka.server.KafkaServer)
>>>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during
>>>>>> KafkaServerStartable
>>>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>>> to
>>>>>> zookeeper server within timeout: 25000
>>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>>> scala:75)
>>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>>> at kafka.server.KafkaServerStarta
>>>>>> ble.startup(KafkaServerStartab
>>>>>> le.scala:37)
>>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>>> [2017-05-17 17:03:40,764] INFO shutting down
>>>>>> (kafka.server.KafkaServer)
>>>>>>
>>>>>> Manually starting Kafka via Ambari fixes this... However, Metron REST
>>>>>> still doesn't work:
>>>>>>
>>>>>> . ____ _ __ _ _
>>>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>>>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>>>>> =========|_|==============|___/=/_/_/_/
>>>>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>>>>
>>>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri
>>>>>> ngApplication]
>>>>>> - Application startup failed
>>>>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>>>>> Error
>>>>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic
>>>>>> eImpl.class]:
>>>>>> Unsatisfied dependency expressed through constructor parameter 0;
>>>>>> nested
>>>>>> exception is org.springframework.beans.fact
>>>>>> ory.UnsatisfiedDependencyException:
>>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>>> expressed through field 'zkClient'; nested exception is
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating
>>>>>> bean with name 'zkClient' defined in class path resource
>>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>>> instantiation
>>>>>> via factory method failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>>> ZkTimeoutException:
>>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.createArgumentArray(ConstructorResolver.java:749)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.autowireConstructor(ConstructorResolver.java:189)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.autowireConstructor(AbstractAu
>>>>>> towireCapableBeanFactory.java:1148)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>>> owireCapableBeanFactory.java:1051)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>>> tory.java:510)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>>> ry.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBe
>>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBea
>>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory
>>>>>> .java:751)
>>>>>> at org.springframework.context.su
>>>>>> pport.AbstractApplicationConte
>>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.jav
>>>>>> a:861)
>>>>>> at org.springframework.context.su
>>>>>> pport.AbstractApplicationConte
>>>>>> xt.refresh(AbstractApplicationContext.java:541)
>>>>>> at org.springframework.boot.conte
>>>>>> xt.embedded.EmbeddedWebApplica
>>>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>>>>> at org.springframework.boot.Sprin
>>>>>> gApplication.refresh(SpringApp
>>>>>> lication.java:761)
>>>>>> at org.springframework.boot.Sprin
>>>>>> gApplication.refreshContext(Sp
>>>>>> ringApplication.java:371)
>>>>>> at org.springframework.boot.Sprin
>>>>>> gApplication.run(SpringApplica
>>>>>> tion.java:315)
>>>>>> at org.springframework.boot.Sprin
>>>>>> gApplication.run(SpringApplica
>>>>>> tion.java:1186)
>>>>>> at org.springframework.boot.Sprin
>>>>>> gApplication.run(SpringApplica
>>>>>> tion.java:1175)
>>>>>> at org.apache.metron.rest.MetronR
>>>>>> estApplication.main(MetronRest
>>>>>> Application.java:27)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>> Method)
>>>>>> at sun.reflect.NativeMethodAccess
>>>>>> orImpl.invoke(NativeMethodAcce
>>>>>> ssorImpl.java:62)
>>>>>> at sun.reflect.DelegatingMethodAc
>>>>>> cessorImpl.invoke(DelegatingMe
>>>>>> thodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at org.springframework.boot.loade
>>>>>> r.MainMethodRunner.run(MainMet
>>>>>> hodRunner.java:48)
>>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>>> java:87)
>>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>>> java:50)
>>>>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>>>>> PropertiesLauncher.java:525)
>>>>>> Caused by: org.springframework.beans.fact
>>>>>> ory.UnsatisfiedDependencyExcep
>>>>>> tion:
>>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>>> expressed through field 'zkClient'; nested exception is
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating
>>>>>> bean with name 'zkClient' defined in class path resource
>>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>>> instantiation
>>>>>> via factory method failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>>> ZkTimeoutException:
>>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>>> at org.springframework.beans.fact
>>>>>> ory.annotation.AutowiredAnnota
>>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>>> utowiredAnnotationBeanPostProcessor.java:569)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.annotation.InjectionMetada
>>>>>> ta.inject(InjectionMetadata.java:88)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.annotation.AutowiredAnnota
>>>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>>>>> otationBeanPostProcessor.java:349)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>>>>> tory.java:1219)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>>> tory.java:543)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>>> ry.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBe
>>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>>> ireCapableBeanFactory.java:1128)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>>> owireCapableBeanFactory.java:1023)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>>> tory.java:510)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>>> ry.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBe
>>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.config.DependencyDescripto
>>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBea
>>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBea
>>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.createArgumentArray(ConstructorResolver.java:741)
>>>>>> ... 27 more
>>>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>>>> Error
>>>>>> creating bean with name 'zkClient' defined in class path resource
>>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>>> instantiation
>>>>>> via factory method failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>>> ZkTimeoutException:
>>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>>> ireCapableBeanFactory.java:1128)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>>> owireCapableBeanFactory.java:1023)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>>> tory.java:510)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCa
>>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>>> ry.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBe
>>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactor
>>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.config.DependencyDescripto
>>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBea
>>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBea
>>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.annotation.AutowiredAnnota
>>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>>> utowiredAnnotationBeanPostProcessor.java:566)
>>>>>> ... 50 more
>>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>>> Failed
>>>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>>> 'zkClient'
>>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>>> ZkTimeoutException:
>>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.SimpleInstantiatio
>>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.ConstructorResolve
>>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>>>>> ... 62 more
>>>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable
>>>>>> to
>>>>>> connect to zookeeper server within timeout: 10000
>>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>>>>> at org.apache.metron.rest.config.
>>>>>> ZookeeperConfig.zkClient(Zooke
>>>>>> eperConfig.java:52)
>>>>>> at org.apache.metron.rest.config.
>>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>>>>> at org.apache.metron.rest.config.
>>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(
>>>>>> <generated>)
>>>>>> at org.springframework.cglib.prox
>>>>>> y.MethodProxy.invokeSuper(Meth
>>>>>> odProxy.java:228)
>>>>>> at org.springframework.context.an
>>>>>> notation.ConfigurationClassEnh
>>>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha
>>>>>> ncer.java:356)
>>>>>> at org.apache.metron.rest.config.
>>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>> Method)
>>>>>> at sun.reflect.NativeMethodAccess
>>>>>> orImpl.invoke(NativeMethodAcce
>>>>>> ssorImpl.java:62)
>>>>>> at sun.reflect.DelegatingMethodAc
>>>>>> cessorImpl.invoke(DelegatingMe
>>>>>> thodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.SimpleInstantiatio
>>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>>>>> ... 63 more
>>>>>>
>>>>>> Any ideas on what might be going or why the timeout triggers?
>>>>>>
>>>>>>
>>>>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>>>>
>>>>>> Hello list,
>>>>>>
>>>>>>>
>>>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>>>>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>>>>> instances instead of 10:
>>>>>>>
>>>>>>> <-- SNIP -->
>>>>>>> tasks:
>>>>>>> - include: tasks/create-keypair.yml
>>>>>>> - include: tasks/create-vpc.yml
>>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>>> # - include: tasks/create-hosts.yml host_count=4
>>>>>>> host_type=ambari_slave,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=pcap_server,monit,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>>> # - include: tasks/create-hosts.yml host_count=2
>>>>>>> host_type=ambari_slave,search,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,search,ec2
>>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>>> host_type=ambari_slave,web,ec2
>>>>>>> <-- END SNIP -->
>>>>>>>
>>>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>>>>> playbook.yml \
>>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>>> --extra-vars="env=$ENV" \
>>>>>>> $EXTRA_ARGS"
>>>>>>>
>>>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>>> --extra-vars="env=$ENV" \
>>>>>>> $EXTRA_ARGS -c paramiko"
>>>>>>> in run.sh otherwise ssh would time out.
>>>>>>>
>>>>>>> However, Ansible errors out with:
>>>>>>>
>>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>>>>> Deploy cluster with Ambari;
>>>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>>> ambari_config/tasks/main.yml:36
>>>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>>>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
>>>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>>>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [],
>>>>>>> "name":
>>>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>>>>> {"es_hosts":
>>>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password":
>>>>>>> "root",
>>>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>>>>> "storm_rest_addr":
>>>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>>>>> "zeppelin_server_url":
>>>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}},
>>>>>>> {"kibana-env":
>>>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>>>>> "kibana_es_url":
>>>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name":
>>>>>>> "HDP",
>>>>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>>>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
>>>>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>>>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>>>>> {"dfs.datanode.data.dir":
>>>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>>>>> "/data1/hadoop/yarn/timeline",
>>>>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>>>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>>>>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas":
>>>>>>> 1,
>>>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>>>>>>> "zen_discovery_ping_unicast_hosts": "[
>>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>>>>>>> exception occurred: Could not create cluster: request code 400,
>>>>>>> request message {\n \"status\" : 400,\n \"message\" :
>>>>>>> \"Invalid Cluster Creation Template:
>>>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>>>>> 'host_count' value\"\n}"}
>>>>>>>
>>>>>>> PLAY RECAP ******************************
>>>>>>> ******************************
>>>>>>> *********
>>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49
>>>>>>> changed=27
>>>>>>> unreachable=0 failed=0
>>>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>>>>>> changed=8
>>>>>>> unreachable=0 failed=0
>>>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>>>>>> changed=31
>>>>>>> unreachable=0 failed=0
>>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>>>>>> changed=27
>>>>>>> unreachable=0 failed=0
>>>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>>>>>> changed=28
>>>>>>> unreachable=0 failed=1
>>>>>>> localhost : ok=29 changed=12 unreachable=0
>>>>>>> failed=0
>>>>>>>
>>>>>>> Any idea what might be going on?
>>>>>>>
>>>>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Laurens Vets <la...@daemon.be>.
Hmmm, deploying Metron in AWS from another EC2 instance seems to work
fine...
I have a fully deployed 10 node Metron install now.
On 2017-05-19 10:52, Nick Allen wrote:
> I am not sure what the issue is. Nothing is jumping out at me.
>
> Weird issues can arise when the machine that you are running the
> Ansible
> deployment from has intermittent connectivity issues to the AWS data
> center.
>
> Is Amazon's us-west-2 data center the one closest to you? You might
> try a
> different/closer data center. You can alter the data center here [1].
>
> [1]
> https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L27-L39
>
> Other than that, you could try standing up a small node in AWS and then
> use
> that to run the Metron deployment from. That always works much more
> smoothly.
>
>
>
>
> On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> Also, it's only 1 out of 10 hosts that failing it seems.
>>
>>
>> On 2017-05-17 13:25, Laurens Vets wrote:
>>
>>> When you say Metron host, do you mean the host I'm pushing from to
>>> ec2
>>> or the master ambari server in ec2?
>>>
>>> On 2017-05-17 11:51, Ryan Merriman wrote:
>>>
>>>> That happens when you don't have the zookeeper url configured
>>>> correctly.
>>>> Can you check the contents of the /etc/sysconfig/metron file on the
>>>> Metron
>>>> host?
>>>>
>>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> For testing purposes, I decided to spin up the default Metron AWS
>>>> config.
>>>>> This resulted in a hang from ansible here:
>>>>>
>>>>> TASK [librdkafka : include] ******************************
>>>>> **********************
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/main.yml:18
>>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml for
>>>>> ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>>
>>>>> TASK [librdkafka : Install prerequisites]
>>>>> ******************************
>>>>> ********
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml:18
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>>>> CONNECTION
>>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>>>> FutureWarning: CTR mode needs counter parameter, not IV
>>>>> self._cipher = factory.new(key, *args, **kwargs)
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask
>>>>> 22 &&
>>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" )
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>>>>> /tmp/tmpwnH61y
>>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh
>>>>> -c
>>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>>>> /usr/bin/python
>>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum;
>>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385
>>>>> 3889508/"
>>>>> > /dev/null 2>&1'"'"''
>>>>>
>>>>> Looking in the machine logs, I see the following for Kafka and
>>>>> Metron
>>>>> REST:
>>>>>
>>>>> Kafka:
>>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>>>> advertised.host.name = null
>>>>> metric.reporters = []
>>>>> quota.producer.default = 9223372036854775807
>>>>> offsets.topic.num.partitions = 50
>>>>> log.flush.interval.messages = 9223372036854775807
>>>>> auto.create.topics.enable = true
>>>>> controller.socket.timeout.ms = 30000
>>>>> log.flush.interval.ms = null
>>>>> principal.builder.class = class
>>>>> org.apache.kafka.common.securi
>>>>> ty.auth.DefaultPrincipalBuilder
>>>>> replica.socket.receive.buffer.bytes = 65536
>>>>> min.insync.replicas = 1
>>>>> replica.fetch.wait.max.ms = 500
>>>>> num.recovery.threads.per.data.dir = 1
>>>>> ssl.keystore.type = JKS
>>>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>>>> default.replication.factor = 1
>>>>> ssl.truststore.password = null
>>>>> log.preallocate = false
>>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>>>> fetch.purgatory.purge.interval.requests = 10000
>>>>> ssl.endpoint.identification.algorithm = null
>>>>> replica.socket.timeout.ms = 30000
>>>>> message.max.bytes = 1000000
>>>>> num.io.threads = 8
>>>>> offsets.commit.required.acks = -1
>>>>> log.flush.offset.checkpoint.interval.ms = 60000
>>>>> delete.topic.enable = false
>>>>> quota.window.size.seconds = 1
>>>>> ssl.truststore.type = JKS
>>>>> offsets.commit.timeout.ms = 5000
>>>>> quota.window.num = 11
>>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>>>> ompute.amazonaws.com:2181
>>>>> authorizer.class.name =
>>>>> num.replica.fetchers = 1
>>>>> log.retention.ms = null
>>>>> log.roll.jitter.hours = 0
>>>>> log.cleaner.enable = true
>>>>> offsets.load.buffer.size = 5242880
>>>>> log.cleaner.delete.retention.ms = 86400000
>>>>> ssl.client.auth = none
>>>>> controlled.shutdown.max.retries = 3
>>>>> queued.max.requests = 500
>>>>> offsets.topic.replication.factor = 3
>>>>> log.cleaner.threads = 1
>>>>> sasl.kerberos.service.name = null
>>>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>>>> socket.request.max.bytes = 104857600
>>>>> ssl.trustmanager.algorithm = PKIX
>>>>> zookeeper.session.timeout.ms = 30000
>>>>> log.retention.bytes = -1
>>>>> log.message.timestamp.type = CreateTime
>>>>> sasl.kerberos.min.time.before.relogin = 60000
>>>>> zookeeper.set.acl = false
>>>>> connections.max.idle.ms = 600000
>>>>> offsets.retention.minutes = 86400000
>>>>> replica.fetch.backoff.ms = 1000
>>>>> inter.broker.protocol.version = 0.10.0-IV1
>>>>> log.retention.hours = 168
>>>>> num.partitions = 1
>>>>> broker.id.generation.enable = true
>>>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>>>> us-west-2.compute.amazonaws.com:6667
>>>>> ssl.provider = null
>>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>>>> log.roll.ms = null
>>>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>>>> ssl.cipher.suites = null
>>>>> log.index.size.max.bytes = 10485760
>>>>> ssl.keymanager.algorithm = SunX509
>>>>> security.inter.broker.protocol = PLAINTEXT
>>>>> replica.fetch.max.bytes = 1048576
>>>>> advertised.port = null
>>>>> log.cleaner.dedupe.buffer.size = 134217728
>>>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>>>> log.cleaner.io.buffer.size = 524288
>>>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>>>> zookeeper.connection.timeout.ms = 25000
>>>>> controlled.shutdown.retry.backoff.ms = 5000
>>>>> log.roll.hours = 168
>>>>> log.cleanup.policy = delete
>>>>> host.name =
>>>>> log.roll.jitter.ms = null
>>>>> max.connections.per.ip = 2147483647
>>>>> offsets.topic.segment.bytes = 104857600
>>>>> background.threads = 10
>>>>> quota.consumer.default = 9223372036854775807
>>>>> request.timeout.ms = 30000
>>>>> log.message.format.version = 0.10.0-IV1
>>>>> log.index.interval.bytes = 4096
>>>>> log.dir = /tmp/kafka-logs
>>>>> log.segment.bytes = 1073741824
>>>>> log.cleaner.backoff.ms = 15000
>>>>> offset.metadata.max.bytes = 4096
>>>>> ssl.truststore.location = null
>>>>> group.max.session.timeout.ms = 300000
>>>>> ssl.keystore.password = null
>>>>> zookeeper.sync.time.ms = 2000
>>>>> port = 6667
>>>>> log.retention.minutes = null
>>>>> log.segment.delete.delay.ms = 60000
>>>>> log.dirs = /data2/kafka-log
>>>>> controlled.shutdown.enable = true
>>>>> compression.type = producer
>>>>> max.connections.per.ip.overrides =
>>>>> log.message.timestamp.difference.max.ms =
>>>>> 9223372036854775807
>>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>>>> log.cleaner.io.max.bytes.per.second =
>>>>> 1.7976931348623157E308
>>>>> auto.leader.rebalance.enable = true
>>>>> leader.imbalance.check.interval.seconds = 300
>>>>> log.cleaner.min.cleanable.ratio = 0.5
>>>>> replica.lag.time.max.ms = 10000
>>>>> num.network.threads = 3
>>>>> ssl.key.password = null
>>>>> reserved.broker.max.id = 1000
>>>>> metrics.num.samples = 2
>>>>> socket.send.buffer.bytes = 102400
>>>>> ssl.protocol = TLS
>>>>> socket.receive.buffer.bytes = 102400
>>>>> ssl.keystore.location = null
>>>>> replica.fetch.min.bytes = 1
>>>>> broker.rack = null
>>>>> unclean.leader.election.enable = true
>>>>> sasl.enabled.mechanisms = [GSSAPI]
>>>>> group.min.session.timeout.ms = 6000
>>>>> log.cleaner.io.buffer.load.factor = 0.9
>>>>> offsets.retention.check.interval.ms = 600000
>>>>> producer.purgatory.purge.interval.requests = 10000
>>>>> metrics.sample.window.ms = 30000
>>>>> broker.id = -1
>>>>> offsets.topic.compression.codec = 0
>>>>> log.retention.check.interval.ms = 300000
>>>>> advertised.listeners = null
>>>>> leader.imbalance.per.broker.percentage = 10
>>>>> (kafka.server.KafkaConfig)
>>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer
>>>>> startup.
>>>>> Prepare to shutdown (kafka.server.KafkaServer)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>> to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at
>>>>> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at
>>>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,740] INFO shutting down
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during
>>>>> KafkaServerStartable
>>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>>>> to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at
>>>>> kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at
>>>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,764] INFO shutting down
>>>>> (kafka.server.KafkaServer)
>>>>>
>>>>> Manually starting Kafka via Ambari fixes this... However, Metron
>>>>> REST
>>>>> still doesn't work:
>>>>>
>>>>> . ____ _ __ _ _
>>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>>>> =========|_|==============|___/=/_/_/_/
>>>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>>>
>>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri
>>>>> ngApplication]
>>>>> - Application startup failed
>>>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>>>> Error
>>>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic
>>>>> eImpl.class]:
>>>>> Unsatisfied dependency expressed through constructor parameter 0;
>>>>> nested
>>>>> exception is org.springframework.beans.fact
>>>>> ory.UnsatisfiedDependencyException:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:749)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.autowireConstructor(ConstructorResolver.java:189)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.autowireConstructor(AbstractAu
>>>>> towireCapableBeanFactory.java:1148)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1051)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>>>>> at
>>>>> org.springframework.context.support.AbstractApplicationConte
>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
>>>>> at
>>>>> org.springframework.context.support.AbstractApplicationConte
>>>>> xt.refresh(AbstractApplicationContext.java:541)
>>>>> at
>>>>> org.springframework.boot.context.embedded.EmbeddedWebApplica
>>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.refresh(SpringApp
>>>>> lication.java:761)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.refreshContext(Sp
>>>>> ringApplication.java:371)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:315)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:1186)
>>>>> at
>>>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>>>> tion.java:1175)
>>>>> at
>>>>> org.apache.metron.rest.MetronRestApplication.main(MetronRest
>>>>> Application.java:27)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.springframework.boot.loader.MainMethodRunner.run(MainMet
>>>>> hodRunner.java:48)
>>>>> at
>>>>> org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:87)
>>>>> at
>>>>> org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:50)
>>>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>>>> PropertiesLauncher.java:525)
>>>>> Caused by:
>>>>> org.springframework.beans.factory.UnsatisfiedDependencyExcep
>>>>> tion:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:569)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.InjectionMetada
>>>>> ta.inject(InjectionMetadata.java:88)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>>>> otationBeanPostProcessor.java:349)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:1219)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:543)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at
>>>>> org.springframework.beans.factory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:741)
>>>>> ... 27 more
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>>> Error
>>>>> creating bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at
>>>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at
>>>>> org.springframework.beans.factory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at
>>>>> org.springframework.beans.factory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at
>>>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:566)
>>>>> ... 50 more
>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>> Failed
>>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at
>>>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>>>> at
>>>>> org.springframework.beans.factory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>>>> ... 62 more
>>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable
>>>>> to
>>>>> connect to zookeeper server within timeout: 10000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
>>>>> eperConfig.java:52)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(
>>>>> <generated>)
>>>>> at
>>>>> org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
>>>>> odProxy.java:228)
>>>>> at
>>>>> org.springframework.context.annotation.ConfigurationClassEnh
>>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha
>>>>> ncer.java:356)
>>>>> at
>>>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>>>> ... 63 more
>>>>>
>>>>> Any ideas on what might be going or why the timeout triggers?
>>>>>
>>>>>
>>>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>>>
>>>>> Hello list,
>>>>>>
>>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made
>>>>>> the
>>>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>>>> instances instead of 10:
>>>>>>
>>>>>> <-- SNIP -->
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>> # - include: tasks/create-hosts.yml host_count=4
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>> # - include: tasks/create-hosts.yml host_count=2
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,web,ec2
>>>>>> <-- END SNIP -->
>>>>>>
>>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>>>> playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS"
>>>>>>
>>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS -c paramiko"
>>>>>> in run.sh otherwise ssh would time out.
>>>>>>
>>>>>> However, Ansible errors out with:
>>>>>>
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>>>> Deploy cluster with Ambari;
>>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> ambari_config/tasks/main.yml:36
>>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name":
>>>>>> "YARN_CLIENT"},
>>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>> "ES_SLAVE"}],
>>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration":
>>>>>> [],
>>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration":
>>>>>> [],
>>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [],
>>>>>> "name":
>>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration":
>>>>>> [],
>>>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>>>> {"es_hosts":
>>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password":
>>>>>> "root",
>>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>>>> "storm_rest_addr":
>>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>>>> "zeppelin_server_url":
>>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}},
>>>>>> {"kibana-env":
>>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>>>> "kibana_es_url":
>>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name":
>>>>>> "HDP",
>>>>>> "stack_version": "2.5"}, "cluster_name": "metron",
>>>>>> "cluster_state":
>>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize":
>>>>>> 1024,
>>>>>> "namenode_heapsize": 2048}}, {"hbase-env":
>>>>>> {"hbase_master_heapsize":
>>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>>>> {"dfs.datanode.data.dir":
>>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>>>> {"mapred-site":
>>>>>> {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>>>> "/data1/hadoop/yarn/timeline",
>>>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site":
>>>>>> {"storm.local.dir":
>>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701,
>>>>>> 6702,
>>>>>> 6703]", "topology.classpath":
>>>>>> "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>>>> {"gateway_recover_after_data_nodes": 1,
>>>>>> "index_number_of_replicas": 1,
>>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_
>>>>>> ]",
>>>>>> "zen_discovery_ping_unicast_hosts": "[
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari
>>>>>> client
>>>>>> exception occurred: Could not create cluster: request code 400,
>>>>>> request message {\n \"status\" : 400,\n \"message\"
>>>>>> :
>>>>>> \"Invalid Cluster Creation Template:
>>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>>>> 'host_count' value\"\n}"}
>>>>>>
>>>>>> PLAY RECAP ******************************
>>>>>> ******************************
>>>>>> *********
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>>>>> changed=8
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>>>>> changed=31
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>>>>> changed=28
>>>>>> unreachable=0 failed=1
>>>>>> localhost : ok=29 changed=12 unreachable=0
>>>>>> failed=0
>>>>>>
>>>>>> Any idea what might be going on?
>>>>>>
>>>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>>>
>>>>>>
>>>>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Nick Allen <ni...@nickallen.org>.
Missed one important point. You actually choose the data center here [2]
in the configuration file.
[2]
https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L43
On Fri, May 19, 2017 at 1:52 PM, Nick Allen <ni...@nickallen.org> wrote:
> I am not sure what the issue is. Nothing is jumping out at me.
>
> Weird issues can arise when the machine that you are running the Ansible
> deployment from has intermittent connectivity issues to the AWS data center.
>
> Is Amazon's us-west-2 data center the one closest to you? You might try a
> different/closer data center. You can alter the data center here [1].
>
> [1] https://github.com/apache/metron/blob/master/metron-
> deployment/amazon-ec2/conf/defaults.yml#L27-L39
>
> Other than that, you could try standing up a small node in AWS and then
> use that to run the Metron deployment from. That always works much more
> smoothly.
>
>
>
>
> On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <la...@daemon.be> wrote:
>
>> Also, it's only 1 out of 10 hosts that failing it seems.
>>
>>
>> On 2017-05-17 13:25, Laurens Vets wrote:
>>
>>> When you say Metron host, do you mean the host I'm pushing from to ec2
>>> or the master ambari server in ec2?
>>>
>>> On 2017-05-17 11:51, Ryan Merriman wrote:
>>>
>>>> That happens when you don't have the zookeeper url configured correctly.
>>>> Can you check the contents of the /etc/sysconfig/metron file on the
>>>> Metron
>>>> host?
>>>>
>>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
>>>> wrote:
>>>>
>>>> For testing purposes, I decided to spin up the default Metron AWS
>>>>> config.
>>>>> This resulted in a hang from ansible here:
>>>>>
>>>>> TASK [librdkafka : include] ******************************
>>>>> **********************
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/main.yml:18
>>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>>
>>>>> TASK [librdkafka : Install prerequisites]
>>>>> ******************************
>>>>> ********
>>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> librdkafka/tasks/dependencies.yml:18
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>>>> CONNECTION
>>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>>>> ompute.amazonaws.com
>>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>>>> FutureWarning: CTR mode needs counter parameter, not IV
>>>>> self._cipher = factory.new(key, *args, **kwargs)
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22
>>>>> &&
>>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>>> -1495041091.74-92163853889508
>>>>> )" )
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>>>>> /tmp/tmpwnH61y
>>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum
>>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
>>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>>>> /usr/bin/python
>>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>>> 889508/yum;
>>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385
>>>>> 3889508/"
>>>>> > /dev/null 2>&1'"'"''
>>>>>
>>>>> Looking in the machine logs, I see the following for Kafka and Metron
>>>>> REST:
>>>>>
>>>>> Kafka:
>>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>>>> advertised.host.name = null
>>>>> metric.reporters = []
>>>>> quota.producer.default = 9223372036854775807
>>>>> offsets.topic.num.partitions = 50
>>>>> log.flush.interval.messages = 9223372036854775807
>>>>> auto.create.topics.enable = true
>>>>> controller.socket.timeout.ms = 30000
>>>>> log.flush.interval.ms = null
>>>>> principal.builder.class = class org.apache.kafka.common.securi
>>>>> ty.auth.DefaultPrincipalBuilder
>>>>> replica.socket.receive.buffer.bytes = 65536
>>>>> min.insync.replicas = 1
>>>>> replica.fetch.wait.max.ms = 500
>>>>> num.recovery.threads.per.data.dir = 1
>>>>> ssl.keystore.type = JKS
>>>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>>>> default.replication.factor = 1
>>>>> ssl.truststore.password = null
>>>>> log.preallocate = false
>>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>>>> fetch.purgatory.purge.interval.requests = 10000
>>>>> ssl.endpoint.identification.algorithm = null
>>>>> replica.socket.timeout.ms = 30000
>>>>> message.max.bytes = 1000000
>>>>> num.io.threads = 8
>>>>> offsets.commit.required.acks = -1
>>>>> log.flush.offset.checkpoint.interval.ms = 60000
>>>>> delete.topic.enable = false
>>>>> quota.window.size.seconds = 1
>>>>> ssl.truststore.type = JKS
>>>>> offsets.commit.timeout.ms = 5000
>>>>> quota.window.num = 11
>>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>>>> ompute.amazonaws.com:2181
>>>>> authorizer.class.name =
>>>>> num.replica.fetchers = 1
>>>>> log.retention.ms = null
>>>>> log.roll.jitter.hours = 0
>>>>> log.cleaner.enable = true
>>>>> offsets.load.buffer.size = 5242880
>>>>> log.cleaner.delete.retention.ms = 86400000
>>>>> ssl.client.auth = none
>>>>> controlled.shutdown.max.retries = 3
>>>>> queued.max.requests = 500
>>>>> offsets.topic.replication.factor = 3
>>>>> log.cleaner.threads = 1
>>>>> sasl.kerberos.service.name = null
>>>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>>>> socket.request.max.bytes = 104857600
>>>>> ssl.trustmanager.algorithm = PKIX
>>>>> zookeeper.session.timeout.ms = 30000
>>>>> log.retention.bytes = -1
>>>>> log.message.timestamp.type = CreateTime
>>>>> sasl.kerberos.min.time.before.relogin = 60000
>>>>> zookeeper.set.acl = false
>>>>> connections.max.idle.ms = 600000
>>>>> offsets.retention.minutes = 86400000
>>>>> replica.fetch.backoff.ms = 1000
>>>>> inter.broker.protocol.version = 0.10.0-IV1
>>>>> log.retention.hours = 168
>>>>> num.partitions = 1
>>>>> broker.id.generation.enable = true
>>>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>>>> us-west-2.compute.amazonaws.com:6667
>>>>> ssl.provider = null
>>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>>>> log.roll.ms = null
>>>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>>>> ssl.cipher.suites = null
>>>>> log.index.size.max.bytes = 10485760
>>>>> ssl.keymanager.algorithm = SunX509
>>>>> security.inter.broker.protocol = PLAINTEXT
>>>>> replica.fetch.max.bytes = 1048576
>>>>> advertised.port = null
>>>>> log.cleaner.dedupe.buffer.size = 134217728
>>>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>>>> log.cleaner.io.buffer.size = 524288
>>>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>>>> zookeeper.connection.timeout.ms = 25000
>>>>> controlled.shutdown.retry.backoff.ms = 5000
>>>>> log.roll.hours = 168
>>>>> log.cleanup.policy = delete
>>>>> host.name =
>>>>> log.roll.jitter.ms = null
>>>>> max.connections.per.ip = 2147483647
>>>>> offsets.topic.segment.bytes = 104857600
>>>>> background.threads = 10
>>>>> quota.consumer.default = 9223372036854775807
>>>>> request.timeout.ms = 30000
>>>>> log.message.format.version = 0.10.0-IV1
>>>>> log.index.interval.bytes = 4096
>>>>> log.dir = /tmp/kafka-logs
>>>>> log.segment.bytes = 1073741824
>>>>> log.cleaner.backoff.ms = 15000
>>>>> offset.metadata.max.bytes = 4096
>>>>> ssl.truststore.location = null
>>>>> group.max.session.timeout.ms = 300000
>>>>> ssl.keystore.password = null
>>>>> zookeeper.sync.time.ms = 2000
>>>>> port = 6667
>>>>> log.retention.minutes = null
>>>>> log.segment.delete.delay.ms = 60000
>>>>> log.dirs = /data2/kafka-log
>>>>> controlled.shutdown.enable = true
>>>>> compression.type = producer
>>>>> max.connections.per.ip.overrides =
>>>>> log.message.timestamp.difference.max.ms = 9223372036854775807
>>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>>>>> auto.leader.rebalance.enable = true
>>>>> leader.imbalance.check.interval.seconds = 300
>>>>> log.cleaner.min.cleanable.ratio = 0.5
>>>>> replica.lag.time.max.ms = 10000
>>>>> num.network.threads = 3
>>>>> ssl.key.password = null
>>>>> reserved.broker.max.id = 1000
>>>>> metrics.num.samples = 2
>>>>> socket.send.buffer.bytes = 102400
>>>>> ssl.protocol = TLS
>>>>> socket.receive.buffer.bytes = 102400
>>>>> ssl.keystore.location = null
>>>>> replica.fetch.min.bytes = 1
>>>>> broker.rack = null
>>>>> unclean.leader.election.enable = true
>>>>> sasl.enabled.mechanisms = [GSSAPI]
>>>>> group.min.session.timeout.ms = 6000
>>>>> log.cleaner.io.buffer.load.factor = 0.9
>>>>> offsets.retention.check.interval.ms = 600000
>>>>> producer.purgatory.purge.interval.requests = 10000
>>>>> metrics.sample.window.ms = 30000
>>>>> broker.id = -1
>>>>> offsets.topic.compression.codec = 0
>>>>> log.retention.check.interval.ms = 300000
>>>>> advertised.listeners = null
>>>>> leader.imbalance.per.broker.percentage = 10
>>>>> (kafka.server.KafkaConfig)
>>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup.
>>>>> Prepare to shutdown (kafka.server.KafkaServer)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at kafka.server.KafkaServerStarta
>>>>> ble.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>>>> (kafka.server.KafkaServer)
>>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable
>>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>>>>> zookeeper server within timeout: 25000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>>> scala:75)
>>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>>> at kafka.server.KafkaServerStarta
>>>>> ble.startup(KafkaServerStartab
>>>>> le.scala:37)
>>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>>> at kafka.Kafka.main(Kafka.scala)
>>>>> [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer)
>>>>>
>>>>> Manually starting Kafka via Ambari fixes this... However, Metron REST
>>>>> still doesn't work:
>>>>>
>>>>> . ____ _ __ _ _
>>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>>>> =========|_|==============|___/=/_/_/_/
>>>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>>>
>>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri
>>>>> ngApplication]
>>>>> - Application startup failed
>>>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>>>> Error
>>>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic
>>>>> eImpl.class]:
>>>>> Unsatisfied dependency expressed through constructor parameter 0;
>>>>> nested
>>>>> exception is org.springframework.beans.fact
>>>>> ory.UnsatisfiedDependencyException:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:749)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.autowireConstructor(ConstructorResolver.java:189)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.autowireConstructor(AbstractAu
>>>>> towireCapableBeanFactory.java:1148)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1051)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBea
>>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>>>>> at org.springframework.context.su
>>>>> pport.AbstractApplicationConte
>>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContex
>>>>> t.java:861)
>>>>> at org.springframework.context.su
>>>>> pport.AbstractApplicationConte
>>>>> xt.refresh(AbstractApplicationContext.java:541)
>>>>> at org.springframework.boot.conte
>>>>> xt.embedded.EmbeddedWebApplica
>>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>>>> at org.springframework.boot.Sprin
>>>>> gApplication.refresh(SpringApp
>>>>> lication.java:761)
>>>>> at org.springframework.boot.Sprin
>>>>> gApplication.refreshContext(Sp
>>>>> ringApplication.java:371)
>>>>> at org.springframework.boot.Sprin
>>>>> gApplication.run(SpringApplica
>>>>> tion.java:315)
>>>>> at org.springframework.boot.Sprin
>>>>> gApplication.run(SpringApplica
>>>>> tion.java:1186)
>>>>> at org.springframework.boot.Sprin
>>>>> gApplication.run(SpringApplica
>>>>> tion.java:1175)
>>>>> at org.apache.metron.rest.MetronR
>>>>> estApplication.main(MetronRest
>>>>> Application.java:27)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccess
>>>>> orImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at sun.reflect.DelegatingMethodAc
>>>>> cessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at org.springframework.boot.loade
>>>>> r.MainMethodRunner.run(MainMet
>>>>> hodRunner.java:48)
>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:87)
>>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>>> java:50)
>>>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>>>> PropertiesLauncher.java:525)
>>>>> Caused by: org.springframework.beans.fact
>>>>> ory.UnsatisfiedDependencyException:
>>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>>> expressed through field 'zkClient'; nested exception is
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating
>>>>> bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at org.springframework.beans.fact
>>>>> ory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:569)
>>>>> at org.springframework.beans.fact
>>>>> ory.annotation.InjectionMetada
>>>>> ta.inject(InjectionMetadata.java:88)
>>>>> at org.springframework.beans.fact
>>>>> ory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>>>> otationBeanPostProcessor.java:349)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:1219)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:543)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:197)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at org.springframework.beans.fact
>>>>> ory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.createArgumentArray(ConstructorResolver.java:741)
>>>>> ... 27 more
>>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>>> Error
>>>>> creating bean with name 'zkClient' defined in class path resource
>>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>>> instantiation
>>>>> via factory method failed; nested exception is
>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>>> ireCapableBeanFactory.java:1128)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>>> owireCapableBeanFactory.java:1023)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>>> tory.java:510)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCa
>>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>>> ry.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBe
>>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactor
>>>>> y.getBean(AbstractBeanFactory.java:202)
>>>>> at org.springframework.beans.fact
>>>>> ory.config.DependencyDescripto
>>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBea
>>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBea
>>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>>> at org.springframework.beans.fact
>>>>> ory.annotation.AutowiredAnnota
>>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>>> utowiredAnnotationBeanPostProcessor.java:566)
>>>>> ... 50 more
>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>> Failed
>>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>>> 'zkClient'
>>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>>> ZkTimeoutException:
>>>>> Unable to connect to zookeeper server within timeout: 10000
>>>>> at org.springframework.beans.fact
>>>>> ory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.ConstructorResolve
>>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>>>> ... 62 more
>>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
>>>>> connect to zookeeper server within timeout: 10000
>>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>>>> at org.apache.metron.rest.config.
>>>>> ZookeeperConfig.zkClient(Zooke
>>>>> eperConfig.java:52)
>>>>> at org.apache.metron.rest.config.
>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>>>> at org.apache.metron.rest.config.
>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(
>>>>> <generated>)
>>>>> at org.springframework.cglib.prox
>>>>> y.MethodProxy.invokeSuper(Meth
>>>>> odProxy.java:228)
>>>>> at org.springframework.context.an
>>>>> notation.ConfigurationClassEnh
>>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha
>>>>> ncer.java:356)
>>>>> at org.apache.metron.rest.config.
>>>>> ZookeeperConfig$$EnhancerBySpr
>>>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccess
>>>>> orImpl.invoke(NativeMethodAcce
>>>>> ssorImpl.java:62)
>>>>> at sun.reflect.DelegatingMethodAc
>>>>> cessorImpl.invoke(DelegatingMe
>>>>> thodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.SimpleInstantiatio
>>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>>>> ... 63 more
>>>>>
>>>>> Any ideas on what might be going or why the timeout triggers?
>>>>>
>>>>>
>>>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>>>
>>>>> Hello list,
>>>>>>
>>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>>>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>>>> instances instead of 10:
>>>>>>
>>>>>> <-- SNIP -->
>>>>>> tasks:
>>>>>> - include: tasks/create-keypair.yml
>>>>>> - include: tasks/create-vpc.yml
>>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>>> # - include: tasks/create-hosts.yml host_count=4
>>>>>> host_type=ambari_slave,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=pcap_server,monit,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>>> # - include: tasks/create-hosts.yml host_count=2
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,search,ec2
>>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>>> host_type=ambari_slave,web,ec2
>>>>>> <-- END SNIP -->
>>>>>>
>>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>>>> playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS"
>>>>>>
>>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>>> --extra-vars="env=$ENV" \
>>>>>> $EXTRA_ARGS -c paramiko"
>>>>>> in run.sh otherwise ssh would time out.
>>>>>>
>>>>>> However, Ansible errors out with:
>>>>>>
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>>>> Deploy cluster with Ambari;
>>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>>> ambari_config/tasks/main.yml:36
>>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
>>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name":
>>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>>>> {"es_hosts":
>>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root",
>>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>>>> "storm_rest_addr":
>>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>>>> "zeppelin_server_url":
>>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
>>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>>>> "kibana_es_url":
>>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP",
>>>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
>>>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>>>> {"dfs.datanode.data.dir":
>>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>>>> "/data1/hadoop/yarn/timeline",
>>>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>>>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas":
>>>>>> 1,
>>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>>>>>> "zen_discovery_ping_unicast_hosts": "[
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>>>>>> exception occurred: Could not create cluster: request code 400,
>>>>>> request message {\n \"status\" : 400,\n \"message\" :
>>>>>> \"Invalid Cluster Creation Template:
>>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>>>> 'host_count' value\"\n}"}
>>>>>>
>>>>>> PLAY RECAP ******************************
>>>>>> ******************************
>>>>>> *********
>>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>>>>> changed=8
>>>>>> unreachable=0 failed=0
>>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>>>>> changed=31
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>>>>> changed=27
>>>>>> unreachable=0 failed=0
>>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>>>>> changed=28
>>>>>> unreachable=0 failed=1
>>>>>> localhost : ok=29 changed=12 unreachable=0
>>>>>> failed=0
>>>>>>
>>>>>> Any idea what might be going on?
>>>>>>
>>>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>>>
>>>>>>
>>>>>
>
Re: Trying to spin up Metron in EC2: Failed
Posted by Nick Allen <ni...@nickallen.org>.
I am not sure what the issue is. Nothing is jumping out at me.
Weird issues can arise when the machine that you are running the Ansible
deployment from has intermittent connectivity issues to the AWS data center.
Is Amazon's us-west-2 data center the one closest to you? You might try a
different/closer data center. You can alter the data center here [1].
[1]
https://github.com/apache/metron/blob/master/metron-deployment/amazon-ec2/conf/defaults.yml#L27-L39
Other than that, you could try standing up a small node in AWS and then use
that to run the Metron deployment from. That always works much more
smoothly.
On Wed, May 17, 2017 at 4:41 PM, Laurens Vets <la...@daemon.be> wrote:
> Also, it's only 1 out of 10 hosts that failing it seems.
>
>
> On 2017-05-17 13:25, Laurens Vets wrote:
>
>> When you say Metron host, do you mean the host I'm pushing from to ec2
>> or the master ambari server in ec2?
>>
>> On 2017-05-17 11:51, Ryan Merriman wrote:
>>
>>> That happens when you don't have the zookeeper url configured correctly.
>>> Can you check the contents of the /etc/sysconfig/metron file on the
>>> Metron
>>> host?
>>>
>>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be> wrote:
>>>
>>> For testing purposes, I decided to spin up the default Metron AWS config.
>>>> This resulted in a hang from ansible here:
>>>>
>>>> TASK [librdkafka : include] ******************************
>>>> **********************
>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>> librdkafka/tasks/main.yml:18
>>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
>>>> ompute.amazonaws.com
>>>>
>>>> TASK [librdkafka : Install prerequisites] ******************************
>>>> ********
>>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>> librdkafka/tasks/dependencies.yml:18
>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>>> CONNECTION
>>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>>> ompute.amazonaws.com
>>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>>> FutureWarning: CTR mode needs counter parameter, not IV
>>>> self._cipher = factory.new(key, *args, **kwargs)
>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 &&
>>>> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>> -1495041091.74-92163853889508
>>>> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp
>>>> -1495041091.74-92163853889508
>>>> )" )
>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT /tmp/tmpwnH61y
>>>> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>> 889508/yum
>>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
>>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>>> /usr/bin/python
>>>> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853
>>>> 889508/yum;
>>>> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-9216385
>>>> 3889508/"
>>>> > /dev/null 2>&1'"'"''
>>>>
>>>> Looking in the machine logs, I see the following for Kafka and Metron
>>>> REST:
>>>>
>>>> Kafka:
>>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>>> advertised.host.name = null
>>>> metric.reporters = []
>>>> quota.producer.default = 9223372036854775807
>>>> offsets.topic.num.partitions = 50
>>>> log.flush.interval.messages = 9223372036854775807
>>>> auto.create.topics.enable = true
>>>> controller.socket.timeout.ms = 30000
>>>> log.flush.interval.ms = null
>>>> principal.builder.class = class org.apache.kafka.common.securi
>>>> ty.auth.DefaultPrincipalBuilder
>>>> replica.socket.receive.buffer.bytes = 65536
>>>> min.insync.replicas = 1
>>>> replica.fetch.wait.max.ms = 500
>>>> num.recovery.threads.per.data.dir = 1
>>>> ssl.keystore.type = JKS
>>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>>> default.replication.factor = 1
>>>> ssl.truststore.password = null
>>>> log.preallocate = false
>>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>>> fetch.purgatory.purge.interval.requests = 10000
>>>> ssl.endpoint.identification.algorithm = null
>>>> replica.socket.timeout.ms = 30000
>>>> message.max.bytes = 1000000
>>>> num.io.threads = 8
>>>> offsets.commit.required.acks = -1
>>>> log.flush.offset.checkpoint.interval.ms = 60000
>>>> delete.topic.enable = false
>>>> quota.window.size.seconds = 1
>>>> ssl.truststore.type = JKS
>>>> offsets.commit.timeout.ms = 5000
>>>> quota.window.num = 11
>>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>>> ompute.amazonaws.com:2181
>>>> authorizer.class.name =
>>>> num.replica.fetchers = 1
>>>> log.retention.ms = null
>>>> log.roll.jitter.hours = 0
>>>> log.cleaner.enable = true
>>>> offsets.load.buffer.size = 5242880
>>>> log.cleaner.delete.retention.ms = 86400000
>>>> ssl.client.auth = none
>>>> controlled.shutdown.max.retries = 3
>>>> queued.max.requests = 500
>>>> offsets.topic.replication.factor = 3
>>>> log.cleaner.threads = 1
>>>> sasl.kerberos.service.name = null
>>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>>> socket.request.max.bytes = 104857600
>>>> ssl.trustmanager.algorithm = PKIX
>>>> zookeeper.session.timeout.ms = 30000
>>>> log.retention.bytes = -1
>>>> log.message.timestamp.type = CreateTime
>>>> sasl.kerberos.min.time.before.relogin = 60000
>>>> zookeeper.set.acl = false
>>>> connections.max.idle.ms = 600000
>>>> offsets.retention.minutes = 86400000
>>>> replica.fetch.backoff.ms = 1000
>>>> inter.broker.protocol.version = 0.10.0-IV1
>>>> log.retention.hours = 168
>>>> num.partitions = 1
>>>> broker.id.generation.enable = true
>>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>>> us-west-2.compute.amazonaws.com:6667
>>>> ssl.provider = null
>>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>>> log.roll.ms = null
>>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>>> ssl.cipher.suites = null
>>>> log.index.size.max.bytes = 10485760
>>>> ssl.keymanager.algorithm = SunX509
>>>> security.inter.broker.protocol = PLAINTEXT
>>>> replica.fetch.max.bytes = 1048576
>>>> advertised.port = null
>>>> log.cleaner.dedupe.buffer.size = 134217728
>>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>>> log.cleaner.io.buffer.size = 524288
>>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>>> zookeeper.connection.timeout.ms = 25000
>>>> controlled.shutdown.retry.backoff.ms = 5000
>>>> log.roll.hours = 168
>>>> log.cleanup.policy = delete
>>>> host.name =
>>>> log.roll.jitter.ms = null
>>>> max.connections.per.ip = 2147483647
>>>> offsets.topic.segment.bytes = 104857600
>>>> background.threads = 10
>>>> quota.consumer.default = 9223372036854775807
>>>> request.timeout.ms = 30000
>>>> log.message.format.version = 0.10.0-IV1
>>>> log.index.interval.bytes = 4096
>>>> log.dir = /tmp/kafka-logs
>>>> log.segment.bytes = 1073741824
>>>> log.cleaner.backoff.ms = 15000
>>>> offset.metadata.max.bytes = 4096
>>>> ssl.truststore.location = null
>>>> group.max.session.timeout.ms = 300000
>>>> ssl.keystore.password = null
>>>> zookeeper.sync.time.ms = 2000
>>>> port = 6667
>>>> log.retention.minutes = null
>>>> log.segment.delete.delay.ms = 60000
>>>> log.dirs = /data2/kafka-log
>>>> controlled.shutdown.enable = true
>>>> compression.type = producer
>>>> max.connections.per.ip.overrides =
>>>> log.message.timestamp.difference.max.ms = 9223372036854775807
>>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>>>> auto.leader.rebalance.enable = true
>>>> leader.imbalance.check.interval.seconds = 300
>>>> log.cleaner.min.cleanable.ratio = 0.5
>>>> replica.lag.time.max.ms = 10000
>>>> num.network.threads = 3
>>>> ssl.key.password = null
>>>> reserved.broker.max.id = 1000
>>>> metrics.num.samples = 2
>>>> socket.send.buffer.bytes = 102400
>>>> ssl.protocol = TLS
>>>> socket.receive.buffer.bytes = 102400
>>>> ssl.keystore.location = null
>>>> replica.fetch.min.bytes = 1
>>>> broker.rack = null
>>>> unclean.leader.election.enable = true
>>>> sasl.enabled.mechanisms = [GSSAPI]
>>>> group.min.session.timeout.ms = 6000
>>>> log.cleaner.io.buffer.load.factor = 0.9
>>>> offsets.retention.check.interval.ms = 600000
>>>> producer.purgatory.purge.interval.requests = 10000
>>>> metrics.sample.window.ms = 30000
>>>> broker.id = -1
>>>> offsets.topic.compression.codec = 0
>>>> log.retention.check.interval.ms = 300000
>>>> advertised.listeners = null
>>>> leader.imbalance.per.broker.percentage = 10
>>>> (kafka.server.KafkaConfig)
>>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>>> (kafka.server.KafkaServer)
>>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup.
>>>> Prepare to shutdown (kafka.server.KafkaServer)
>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>>>> zookeeper server within timeout: 25000
>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>> scala:75)
>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>> le.scala:37)
>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>> at kafka.Kafka.main(Kafka.scala)
>>>> [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer)
>>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>>> (kafka.server.KafkaServer)
>>>> [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable
>>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>>>> zookeeper server within timeout: 25000
>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>>> scala:75)
>>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>>> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>>> le.scala:37)
>>>> at kafka.Kafka$.main(Kafka.scala:67)
>>>> at kafka.Kafka.main(Kafka.scala)
>>>> [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer)
>>>>
>>>> Manually starting Kafka via Ambari fixes this... However, Metron REST
>>>> still doesn't work:
>>>>
>>>> . ____ _ __ _ _
>>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>>> =========|_|==============|___/=/_/_/_/
>>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>>
>>>> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.Spri
>>>> ngApplication]
>>>> - Application startup failed
>>>> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServic
>>>> eImpl.class]:
>>>> Unsatisfied dependency expressed through constructor parameter 0; nested
>>>> exception is org.springframework.beans.fact
>>>> ory.UnsatisfiedDependencyException:
>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>> expressed through field 'zkClient'; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>> bean with name 'zkClient' defined in class path resource
>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>> instantiation
>>>> via factory method failed; nested exception is
>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>> ZkTimeoutException:
>>>> Unable to connect to zookeeper server within timeout: 10000
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.createArgumentArray(ConstructorResolver.java:749)
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.autowireConstructor(ConstructorResolver.java:189)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.autowireConstructor(AbstractAu
>>>> towireCapableBeanFactory.java:1148)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>> owireCapableBeanFactory.java:1051)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:510)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:197)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>>>> at org.springframework.context.support.AbstractApplicationConte
>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
>>>> at org.springframework.context.support.AbstractApplicationConte
>>>> xt.refresh(AbstractApplicationContext.java:541)
>>>> at org.springframework.boot.context.embedded.EmbeddedWebApplica
>>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>>> at org.springframework.boot.SpringApplication.refresh(SpringApp
>>>> lication.java:761)
>>>> at org.springframework.boot.SpringApplication.refreshContext(Sp
>>>> ringApplication.java:371)
>>>> at org.springframework.boot.SpringApplication.run(SpringApplica
>>>> tion.java:315)
>>>> at org.springframework.boot.SpringApplication.run(SpringApplica
>>>> tion.java:1186)
>>>> at org.springframework.boot.SpringApplication.run(SpringApplica
>>>> tion.java:1175)
>>>> at org.apache.metron.rest.MetronRestApplication.main(MetronRest
>>>> Application.java:27)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.springframework.boot.loader.MainMethodRunner.run(MainMet
>>>> hodRunner.java:48)
>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>> java:87)
>>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>>> java:50)
>>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>>> PropertiesLauncher.java:525)
>>>> Caused by: org.springframework.beans.factory.UnsatisfiedDependencyExcep
>>>> tion:
>>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>>> expressed through field 'zkClient'; nested exception is
>>>> org.springframework.beans.factory.BeanCreationException: Error creating
>>>> bean with name 'zkClient' defined in class path resource
>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>> instantiation
>>>> via factory method failed; nested exception is
>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>> ZkTimeoutException:
>>>> Unable to connect to zookeeper server within timeout: 10000
>>>> at org.springframework.beans.factory.annotation.AutowiredAnnota
>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>> utowiredAnnotationBeanPostProcessor.java:569)
>>>> at org.springframework.beans.factory.annotation.InjectionMetada
>>>> ta.inject(InjectionMetadata.java:88)
>>>> at org.springframework.beans.factory.annotation.AutowiredAnnota
>>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>>> otationBeanPostProcessor.java:349)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:1219)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:543)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:197)
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>> ireCapableBeanFactory.java:1128)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>> owireCapableBeanFactory.java:1023)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:510)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:202)
>>>> at org.springframework.beans.factory.config.DependencyDescripto
>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.createArgumentArray(ConstructorResolver.java:741)
>>>> ... 27 more
>>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>>> Error
>>>> creating bean with name 'zkClient' defined in class path resource
>>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>>> instantiation
>>>> via factory method failed; nested exception is
>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>> ZkTimeoutException:
>>>> Unable to connect to zookeeper server within timeout: 10000
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>>> ireCapableBeanFactory.java:1128)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBeanInstance(AbstractAut
>>>> owireCapableBeanFactory.java:1023)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:510)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:202)
>>>> at org.springframework.beans.factory.config.DependencyDescripto
>>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>>> at org.springframework.beans.factory.annotation.AutowiredAnnota
>>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>>> utowiredAnnotationBeanPostProcessor.java:566)
>>>> ... 50 more
>>>> Caused by: org.springframework.beans.BeanInstantiationException: Failed
>>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>>> 'zkClient'
>>>> threw exception; nested exception is org.I0Itec.zkclient.exception.
>>>> ZkTimeoutException:
>>>> Unable to connect to zookeeper server within timeout: 10000
>>>> at org.springframework.beans.factory.support.SimpleInstantiatio
>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>>> at org.springframework.beans.factory.support.ConstructorResolve
>>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>>> ... 62 more
>>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
>>>> connect to zookeeper server within timeout: 10000
>>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>>> at org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
>>>> eperConfig.java:52)
>>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(
>>>> <generated>)
>>>> at org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
>>>> odProxy.java:228)
>>>> at org.springframework.context.annotation.ConfigurationClassEnh
>>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnha
>>>> ncer.java:356)
>>>> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.springframework.beans.factory.support.SimpleInstantiatio
>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>>> ... 63 more
>>>>
>>>> Any ideas on what might be going or why the timeout triggers?
>>>>
>>>>
>>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>>
>>>> Hello list,
>>>>>
>>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>>> instances instead of 10:
>>>>>
>>>>> <-- SNIP -->
>>>>> tasks:
>>>>> - include: tasks/create-keypair.yml
>>>>> - include: tasks/create-vpc.yml
>>>>> - include: tasks/create-open-inbound-security-group.yml
>>>>> - include: tasks/create-open-outbound-security-group.yml
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=sensors,ambari_master,ec2,monit
>>>>> # - include: tasks/create-hosts.yml host_count=4
>>>>> host_type=ambari_slave,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=pcap_server,monit,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>>> # - include: tasks/create-hosts.yml host_count=2
>>>>> host_type=ambari_slave,search,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,search,ec2
>>>>> - include: tasks/create-hosts.yml host_count=1
>>>>> host_type=ambari_slave,web,ec2
>>>>> <-- END SNIP -->
>>>>>
>>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>>> playbook.yml \
>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>> --extra-vars="env=$ENV" \
>>>>> $EXTRA_ARGS"
>>>>>
>>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>>> --extra-vars="env=$ENV" \
>>>>> $EXTRA_ARGS -c paramiko"
>>>>> in run.sh otherwise ssh would time out.
>>>>>
>>>>> However, Ansible errors out with:
>>>>>
>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>>> Deploy cluster with Ambari;
>>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>>> ambari_config/tasks/main.yml:36
>>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
>>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name":
>>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>>> {"es_hosts":
>>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root",
>>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>>> "storm_rest_addr":
>>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>>> "zeppelin_server_url":
>>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
>>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>>> "kibana_es_url":
>>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP",
>>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
>>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>>> {"dfs.datanode.data.dir":
>>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>>> "/data1/hadoop/yarn/timeline",
>>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1,
>>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>>>>> "zen_discovery_ping_unicast_hosts": "[
>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>>>>> exception occurred: Could not create cluster: request code 400,
>>>>> request message {\n \"status\" : 400,\n \"message\" :
>>>>> \"Invalid Cluster Creation Template:
>>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>>> 'host_count' value\"\n}"}
>>>>>
>>>>> PLAY RECAP ******************************
>>>>> ******************************
>>>>> *********
>>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8
>>>>> unreachable=0 failed=0
>>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31
>>>>> unreachable=0 failed=0
>>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27
>>>>> unreachable=0 failed=0
>>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28
>>>>> unreachable=0 failed=1
>>>>> localhost : ok=29 changed=12 unreachable=0
>>>>> failed=0
>>>>>
>>>>> Any idea what might be going on?
>>>>>
>>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>>
>>>>>
>>>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Laurens Vets <la...@daemon.be>.
Also, it's only 1 out of 10 hosts that failing it seems.
On 2017-05-17 13:25, Laurens Vets wrote:
> When you say Metron host, do you mean the host I'm pushing from to ec2
> or the master ambari server in ec2?
>
> On 2017-05-17 11:51, Ryan Merriman wrote:
>> That happens when you don't have the zookeeper url configured
>> correctly.
>> Can you check the contents of the /etc/sysconfig/metron file on the
>> Metron
>> host?
>>
>> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
>> wrote:
>>
>>> For testing purposes, I decided to spin up the default Metron AWS
>>> config.
>>> This resulted in a hang from ansible here:
>>>
>>> TASK [librdkafka : include] ******************************
>>> **********************
>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>> librdkafka/tasks/main.yml:18
>>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
>>> ompute.amazonaws.com
>>>
>>> TASK [librdkafka : Install prerequisites]
>>> ******************************
>>> ********
>>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>>> librdkafka/tasks/dependencies.yml:18
>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>>> CONNECTION
>>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>>> ompute.amazonaws.com
>>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>>> FutureWarning: CTR mode needs counter parameter, not IV
>>> self._cipher = factory.new(key, *args, **kwargs)
>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22
>>> &&
>>> mkdir -p "$( echo
>>> $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
>>> )" && echo "$( echo
>>> $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
>>> )" )
>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>>> /tmp/tmpwnH61y
>>> TO
>>> /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum
>>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
>>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>>> /usr/bin/python
>>> -tt
>>> /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum;
>>> rm -rf
>>> "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/"
>>> > /dev/null 2>&1'"'"''
>>>
>>> Looking in the machine logs, I see the following for Kafka and Metron
>>> REST:
>>>
>>> Kafka:
>>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>>> advertised.host.name = null
>>> metric.reporters = []
>>> quota.producer.default = 9223372036854775807
>>> offsets.topic.num.partitions = 50
>>> log.flush.interval.messages = 9223372036854775807
>>> auto.create.topics.enable = true
>>> controller.socket.timeout.ms = 30000
>>> log.flush.interval.ms = null
>>> principal.builder.class = class
>>> org.apache.kafka.common.securi
>>> ty.auth.DefaultPrincipalBuilder
>>> replica.socket.receive.buffer.bytes = 65536
>>> min.insync.replicas = 1
>>> replica.fetch.wait.max.ms = 500
>>> num.recovery.threads.per.data.dir = 1
>>> ssl.keystore.type = JKS
>>> sasl.mechanism.inter.broker.protocol = GSSAPI
>>> default.replication.factor = 1
>>> ssl.truststore.password = null
>>> log.preallocate = false
>>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>>> fetch.purgatory.purge.interval.requests = 10000
>>> ssl.endpoint.identification.algorithm = null
>>> replica.socket.timeout.ms = 30000
>>> message.max.bytes = 1000000
>>> num.io.threads = 8
>>> offsets.commit.required.acks = -1
>>> log.flush.offset.checkpoint.interval.ms = 60000
>>> delete.topic.enable = false
>>> quota.window.size.seconds = 1
>>> ssl.truststore.type = JKS
>>> offsets.commit.timeout.ms = 5000
>>> quota.window.num = 11
>>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>>> ompute.amazonaws.com:2181
>>> authorizer.class.name =
>>> num.replica.fetchers = 1
>>> log.retention.ms = null
>>> log.roll.jitter.hours = 0
>>> log.cleaner.enable = true
>>> offsets.load.buffer.size = 5242880
>>> log.cleaner.delete.retention.ms = 86400000
>>> ssl.client.auth = none
>>> controlled.shutdown.max.retries = 3
>>> queued.max.requests = 500
>>> offsets.topic.replication.factor = 3
>>> log.cleaner.threads = 1
>>> sasl.kerberos.service.name = null
>>> sasl.kerberos.ticket.renew.jitter = 0.05
>>> socket.request.max.bytes = 104857600
>>> ssl.trustmanager.algorithm = PKIX
>>> zookeeper.session.timeout.ms = 30000
>>> log.retention.bytes = -1
>>> log.message.timestamp.type = CreateTime
>>> sasl.kerberos.min.time.before.relogin = 60000
>>> zookeeper.set.acl = false
>>> connections.max.idle.ms = 600000
>>> offsets.retention.minutes = 86400000
>>> replica.fetch.backoff.ms = 1000
>>> inter.broker.protocol.version = 0.10.0-IV1
>>> log.retention.hours = 168
>>> num.partitions = 1
>>> broker.id.generation.enable = true
>>> listeners = PLAINTEXT://ec2-34-209-53-166.
>>> us-west-2.compute.amazonaws.com:6667
>>> ssl.provider = null
>>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>>> log.roll.ms = null
>>> log.flush.scheduler.interval.ms = 9223372036854775807
>>> ssl.cipher.suites = null
>>> log.index.size.max.bytes = 10485760
>>> ssl.keymanager.algorithm = SunX509
>>> security.inter.broker.protocol = PLAINTEXT
>>> replica.fetch.max.bytes = 1048576
>>> advertised.port = null
>>> log.cleaner.dedupe.buffer.size = 134217728
>>> replica.high.watermark.checkpoint.interval.ms = 5000
>>> log.cleaner.io.buffer.size = 524288
>>> sasl.kerberos.ticket.renew.window.factor = 0.8
>>> zookeeper.connection.timeout.ms = 25000
>>> controlled.shutdown.retry.backoff.ms = 5000
>>> log.roll.hours = 168
>>> log.cleanup.policy = delete
>>> host.name =
>>> log.roll.jitter.ms = null
>>> max.connections.per.ip = 2147483647
>>> offsets.topic.segment.bytes = 104857600
>>> background.threads = 10
>>> quota.consumer.default = 9223372036854775807
>>> request.timeout.ms = 30000
>>> log.message.format.version = 0.10.0-IV1
>>> log.index.interval.bytes = 4096
>>> log.dir = /tmp/kafka-logs
>>> log.segment.bytes = 1073741824
>>> log.cleaner.backoff.ms = 15000
>>> offset.metadata.max.bytes = 4096
>>> ssl.truststore.location = null
>>> group.max.session.timeout.ms = 300000
>>> ssl.keystore.password = null
>>> zookeeper.sync.time.ms = 2000
>>> port = 6667
>>> log.retention.minutes = null
>>> log.segment.delete.delay.ms = 60000
>>> log.dirs = /data2/kafka-log
>>> controlled.shutdown.enable = true
>>> compression.type = producer
>>> max.connections.per.ip.overrides =
>>> log.message.timestamp.difference.max.ms = 9223372036854775807
>>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>>> auto.leader.rebalance.enable = true
>>> leader.imbalance.check.interval.seconds = 300
>>> log.cleaner.min.cleanable.ratio = 0.5
>>> replica.lag.time.max.ms = 10000
>>> num.network.threads = 3
>>> ssl.key.password = null
>>> reserved.broker.max.id = 1000
>>> metrics.num.samples = 2
>>> socket.send.buffer.bytes = 102400
>>> ssl.protocol = TLS
>>> socket.receive.buffer.bytes = 102400
>>> ssl.keystore.location = null
>>> replica.fetch.min.bytes = 1
>>> broker.rack = null
>>> unclean.leader.election.enable = true
>>> sasl.enabled.mechanisms = [GSSAPI]
>>> group.min.session.timeout.ms = 6000
>>> log.cleaner.io.buffer.load.factor = 0.9
>>> offsets.retention.check.interval.ms = 600000
>>> producer.purgatory.purge.interval.requests = 10000
>>> metrics.sample.window.ms = 30000
>>> broker.id = -1
>>> offsets.topic.compression.codec = 0
>>> log.retention.check.interval.ms = 300000
>>> advertised.listeners = null
>>> leader.imbalance.per.broker.percentage = 10
>>> (kafka.server.KafkaConfig)
>>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>>> (kafka.server.KafkaServer)
>>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer
>>> startup.
>>> Prepare to shutdown (kafka.server.KafkaServer)
>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>> to
>>> zookeeper server within timeout: 25000
>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>> scala:75)
>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>> at
>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>> le.scala:37)
>>> at kafka.Kafka$.main(Kafka.scala:67)
>>> at kafka.Kafka.main(Kafka.scala)
>>> [2017-05-17 17:03:40,740] INFO shutting down
>>> (kafka.server.KafkaServer)
>>> [2017-05-17 17:03:40,747] INFO shut down completed
>>> (kafka.server.KafkaServer)
>>> [2017-05-17 17:03:40,748] FATAL Fatal error during
>>> KafkaServerStartable
>>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
>>> to
>>> zookeeper server within timeout: 25000
>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>>> scala:75)
>>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>>> at
>>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>>> le.scala:37)
>>> at kafka.Kafka$.main(Kafka.scala:67)
>>> at kafka.Kafka.main(Kafka.scala)
>>> [2017-05-17 17:03:40,764] INFO shutting down
>>> (kafka.server.KafkaServer)
>>>
>>> Manually starting Kafka via Ambari fixes this... However, Metron REST
>>> still doesn't work:
>>>
>>> . ____ _ __ _ _
>>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>>> ' |____| .__|_| |_|_| |_\__, | / / / /
>>> =========|_|==============|___/=/_/_/_/
>>> :: Spring Boot :: (v1.4.1.RELEASE)
>>>
>>> 2017-05-17T17:03:52.406 ERROR
>>> [org.springframework.boot.SpringApplication]
>>> - Application startup failed
>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>> Error
>>> creating bean with name 'kafkaServiceImpl' defined in URL
>>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
>>> Unsatisfied dependency expressed through constructor parameter 0;
>>> nested
>>> exception is
>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>> expressed through field 'zkClient'; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error
>>> creating
>>> bean with name 'zkClient' defined in class path resource
>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>> instantiation
>>> via factory method failed; nested exception is
>>> org.springframework.beans.BeanInstantiationException: Failed to
>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>> threw exception; nested exception is
>>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>>> Unable to connect to zookeeper server within timeout: 10000
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.createArgumentArray(ConstructorResolver.java:749)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.autowireConstructor(ConstructorResolver.java:189)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.autowireConstructor(AbstractAu
>>> towireCapableBeanFactory.java:1148)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBeanInstance(AbstractAut
>>> owireCapableBeanFactory.java:1051)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.getBean(AbstractBeanFactory.java:197)
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBea
>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>>> at
>>> org.springframework.context.support.AbstractApplicationConte
>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
>>> at
>>> org.springframework.context.support.AbstractApplicationConte
>>> xt.refresh(AbstractApplicationContext.java:541)
>>> at
>>> org.springframework.boot.context.embedded.EmbeddedWebApplica
>>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>>> at
>>> org.springframework.boot.SpringApplication.refresh(SpringApp
>>> lication.java:761)
>>> at
>>> org.springframework.boot.SpringApplication.refreshContext(Sp
>>> ringApplication.java:371)
>>> at
>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>> tion.java:315)
>>> at
>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>> tion.java:1186)
>>> at
>>> org.springframework.boot.SpringApplication.run(SpringApplica
>>> tion.java:1175)
>>> at
>>> org.apache.metron.rest.MetronRestApplication.main(MetronRest
>>> Application.java:27)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.springframework.boot.loader.MainMethodRunner.run(MainMet
>>> hodRunner.java:48)
>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>> java:87)
>>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>>> java:50)
>>> at org.springframework.boot.loader.PropertiesLauncher.main(
>>> PropertiesLauncher.java:525)
>>> Caused by:
>>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>>> expressed through field 'zkClient'; nested exception is
>>> org.springframework.beans.factory.BeanCreationException: Error
>>> creating
>>> bean with name 'zkClient' defined in class path resource
>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>> instantiation
>>> via factory method failed; nested exception is
>>> org.springframework.beans.BeanInstantiationException: Failed to
>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>> threw exception; nested exception is
>>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>>> Unable to connect to zookeeper server within timeout: 10000
>>> at
>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>> utowiredAnnotationBeanPostProcessor.java:569)
>>> at
>>> org.springframework.beans.factory.annotation.InjectionMetada
>>> ta.inject(InjectionMetadata.java:88)
>>> at
>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>>> otationBeanPostProcessor.java:349)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>>> tory.java:1219)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.getBean(AbstractBeanFactory.java:197)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>> ireCapableBeanFactory.java:1128)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBeanInstance(AbstractAut
>>> owireCapableBeanFactory.java:1023)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.getBean(AbstractBeanFactory.java:202)
>>> at
>>> org.springframework.beans.factory.config.DependencyDescripto
>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBea
>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBea
>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.createArgumentArray(ConstructorResolver.java:741)
>>> ... 27 more
>>> Caused by: org.springframework.beans.factory.BeanCreationException:
>>> Error
>>> creating bean with name 'zkClient' defined in class path resource
>>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>>> instantiation
>>> via factory method failed; nested exception is
>>> org.springframework.beans.BeanInstantiationException: Failed to
>>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>>> threw exception; nested exception is
>>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>>> Unable to connect to zookeeper server within timeout: 10000
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>>> ireCapableBeanFactory.java:1128)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBeanInstance(AbstractAut
>>> owireCapableBeanFactory.java:1023)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>>> at
>>> org.springframework.beans.factory.support.AbstractAutowireCa
>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y$1.getObject(AbstractBeanFactory.java:306)
>>> at
>>> org.springframework.beans.factory.support.DefaultSingletonBe
>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.doGetBean(AbstractBeanFactory.java:302)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactor
>>> y.getBean(AbstractBeanFactory.java:202)
>>> at
>>> org.springframework.beans.factory.config.DependencyDescripto
>>> r.resolveCandidate(DependencyDescriptor.java:207)
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBea
>>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBea
>>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>>> at
>>> org.springframework.beans.factory.annotation.AutowiredAnnota
>>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>>> utowiredAnnotationBeanPostProcessor.java:566)
>>> ... 50 more
>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>> Failed
>>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>>> 'zkClient'
>>> threw exception; nested exception is
>>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>>> Unable to connect to zookeeper server within timeout: 10000
>>> at
>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>>> at
>>> org.springframework.beans.factory.support.ConstructorResolve
>>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>>> ... 62 more
>>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable
>>> to
>>> connect to zookeeper server within timeout: 10000
>>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>>> at
>>> org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
>>> eperConfig.java:52)
>>> at
>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>>> at
>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(<generated>)
>>> at
>>> org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
>>> odProxy.java:228)
>>> at
>>> org.springframework.context.annotation.ConfigurationClassEnh
>>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
>>> at
>>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.springframework.beans.factory.support.SimpleInstantiatio
>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>>> ... 63 more
>>>
>>> Any ideas on what might be going or why the timeout triggers?
>>>
>>>
>>> On 2017-05-17 08:31, Laurens Vets wrote:
>>>
>>>> Hello list,
>>>>
>>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>>>> following changes in playbook.yml so that I'm left with 5 EC2
>>>> instances instead of 10:
>>>>
>>>> <-- SNIP -->
>>>> tasks:
>>>> - include: tasks/create-keypair.yml
>>>> - include: tasks/create-vpc.yml
>>>> - include: tasks/create-open-inbound-security-group.yml
>>>> - include: tasks/create-open-outbound-security-group.yml
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=sensors,ambari_master,ec2,monit
>>>> # - include: tasks/create-hosts.yml host_count=4
>>>> host_type=ambari_slave,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=pcap_server,monit,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>>> # - include: tasks/create-hosts.yml host_count=2
>>>> host_type=ambari_slave,search,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,search,ec2
>>>> - include: tasks/create-hosts.yml host_count=1
>>>> host_type=ambari_slave,web,ec2
>>>> <-- END SNIP -->
>>>>
>>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>>> playbook.yml \
>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>> --extra-vars="env=$ENV" \
>>>> $EXTRA_ARGS"
>>>>
>>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>>> --skip-tags="quick_dev,sensor-stubs" \
>>>> --extra-vars="env=$ENV" \
>>>> $EXTRA_ARGS -c paramiko"
>>>> in run.sh otherwise ssh would time out.
>>>>
>>>> However, Ansible errors out with:
>>>>
>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>>> Deploy cluster with Ambari;
>>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>>> ambari_config/tasks/main.yml:36
>>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>> "ES_SLAVE"}],
>>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [],
>>>> "name":
>>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>>> {"es_hosts":
>>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password":
>>>> "root",
>>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>>> "storm_rest_addr":
>>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>>> "zeppelin_server_url":
>>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}},
>>>> {"kibana-env":
>>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>>> "kibana_es_url":
>>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name":
>>>> "HDP",
>>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize":
>>>> 1024,
>>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>>> {"dfs.datanode.data.dir":
>>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>>> "yarn.timeline-service.leveldb-state-store.path":
>>>> "/data1/hadoop/yarn/timeline",
>>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas":
>>>> 1,
>>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>>>> "zen_discovery_ping_unicast_hosts": "[
>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>>>> exception occurred: Could not create cluster: request code 400,
>>>> request message {\n \"status\" : 400,\n \"message\" :
>>>> \"Invalid Cluster Creation Template:
>>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>>> 'host_count' value\"\n}"}
>>>>
>>>> PLAY RECAP
>>>> ************************************************************
>>>> *********
>>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>>> changed=8
>>>> unreachable=0 failed=0
>>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>>> changed=31
>>>> unreachable=0 failed=0
>>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>>> changed=27
>>>> unreachable=0 failed=0
>>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>>> changed=28
>>>> unreachable=0 failed=1
>>>> localhost : ok=29 changed=12 unreachable=0
>>>> failed=0
>>>>
>>>> Any idea what might be going on?
>>>>
>>>> (Don't worry about the URLs, the instances have been terminated)
>>>>
>>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Laurens Vets <la...@daemon.be>.
When you say Metron host, do you mean the host I'm pushing from to ec2
or the master ambari server in ec2?
On 2017-05-17 11:51, Ryan Merriman wrote:
> That happens when you don't have the zookeeper url configured
> correctly.
> Can you check the contents of the /etc/sysconfig/metron file on the
> Metron
> host?
>
> On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be>
> wrote:
>
>> For testing purposes, I decided to spin up the default Metron AWS
>> config.
>> This resulted in a hang from ansible here:
>>
>> TASK [librdkafka : include] ******************************
>> **********************
>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>> librdkafka/tasks/main.yml:18
>> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
>> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
>> ompute.amazonaws.com
>>
>> TASK [librdkafka : Install prerequisites]
>> ******************************
>> ********
>> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
>> librdkafka/tasks/dependencies.yml:18
>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
>> CONNECTION
>> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
>> ompute.amazonaws.com
>> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
>> FutureWarning: CTR mode needs counter parameter, not IV
>> self._cipher = factory.new(key, *args, **kwargs)
>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22
>> &&
>> mkdir -p "$( echo
>> $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
>> )" && echo "$( echo
>> $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
>> )" )
>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT
>> /tmp/tmpwnH61y
>> TO
>> /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum
>> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
>> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo
>> BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
>> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
>> /usr/bin/python
>> -tt
>> /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum;
>> rm -rf
>> "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/"
>> > /dev/null 2>&1'"'"''
>>
>> Looking in the machine logs, I see the following for Kafka and Metron
>> REST:
>>
>> Kafka:
>> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
>> advertised.host.name = null
>> metric.reporters = []
>> quota.producer.default = 9223372036854775807
>> offsets.topic.num.partitions = 50
>> log.flush.interval.messages = 9223372036854775807
>> auto.create.topics.enable = true
>> controller.socket.timeout.ms = 30000
>> log.flush.interval.ms = null
>> principal.builder.class = class org.apache.kafka.common.securi
>> ty.auth.DefaultPrincipalBuilder
>> replica.socket.receive.buffer.bytes = 65536
>> min.insync.replicas = 1
>> replica.fetch.wait.max.ms = 500
>> num.recovery.threads.per.data.dir = 1
>> ssl.keystore.type = JKS
>> sasl.mechanism.inter.broker.protocol = GSSAPI
>> default.replication.factor = 1
>> ssl.truststore.password = null
>> log.preallocate = false
>> sasl.kerberos.principal.to.local.rules = [DEFAULT]
>> fetch.purgatory.purge.interval.requests = 10000
>> ssl.endpoint.identification.algorithm = null
>> replica.socket.timeout.ms = 30000
>> message.max.bytes = 1000000
>> num.io.threads = 8
>> offsets.commit.required.acks = -1
>> log.flush.offset.checkpoint.interval.ms = 60000
>> delete.topic.enable = false
>> quota.window.size.seconds = 1
>> ssl.truststore.type = JKS
>> offsets.commit.timeout.ms = 5000
>> quota.window.num = 11
>> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
>> ompute.amazonaws.com:2181
>> authorizer.class.name =
>> num.replica.fetchers = 1
>> log.retention.ms = null
>> log.roll.jitter.hours = 0
>> log.cleaner.enable = true
>> offsets.load.buffer.size = 5242880
>> log.cleaner.delete.retention.ms = 86400000
>> ssl.client.auth = none
>> controlled.shutdown.max.retries = 3
>> queued.max.requests = 500
>> offsets.topic.replication.factor = 3
>> log.cleaner.threads = 1
>> sasl.kerberos.service.name = null
>> sasl.kerberos.ticket.renew.jitter = 0.05
>> socket.request.max.bytes = 104857600
>> ssl.trustmanager.algorithm = PKIX
>> zookeeper.session.timeout.ms = 30000
>> log.retention.bytes = -1
>> log.message.timestamp.type = CreateTime
>> sasl.kerberos.min.time.before.relogin = 60000
>> zookeeper.set.acl = false
>> connections.max.idle.ms = 600000
>> offsets.retention.minutes = 86400000
>> replica.fetch.backoff.ms = 1000
>> inter.broker.protocol.version = 0.10.0-IV1
>> log.retention.hours = 168
>> num.partitions = 1
>> broker.id.generation.enable = true
>> listeners = PLAINTEXT://ec2-34-209-53-166.
>> us-west-2.compute.amazonaws.com:6667
>> ssl.provider = null
>> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
>> log.roll.ms = null
>> log.flush.scheduler.interval.ms = 9223372036854775807
>> ssl.cipher.suites = null
>> log.index.size.max.bytes = 10485760
>> ssl.keymanager.algorithm = SunX509
>> security.inter.broker.protocol = PLAINTEXT
>> replica.fetch.max.bytes = 1048576
>> advertised.port = null
>> log.cleaner.dedupe.buffer.size = 134217728
>> replica.high.watermark.checkpoint.interval.ms = 5000
>> log.cleaner.io.buffer.size = 524288
>> sasl.kerberos.ticket.renew.window.factor = 0.8
>> zookeeper.connection.timeout.ms = 25000
>> controlled.shutdown.retry.backoff.ms = 5000
>> log.roll.hours = 168
>> log.cleanup.policy = delete
>> host.name =
>> log.roll.jitter.ms = null
>> max.connections.per.ip = 2147483647
>> offsets.topic.segment.bytes = 104857600
>> background.threads = 10
>> quota.consumer.default = 9223372036854775807
>> request.timeout.ms = 30000
>> log.message.format.version = 0.10.0-IV1
>> log.index.interval.bytes = 4096
>> log.dir = /tmp/kafka-logs
>> log.segment.bytes = 1073741824
>> log.cleaner.backoff.ms = 15000
>> offset.metadata.max.bytes = 4096
>> ssl.truststore.location = null
>> group.max.session.timeout.ms = 300000
>> ssl.keystore.password = null
>> zookeeper.sync.time.ms = 2000
>> port = 6667
>> log.retention.minutes = null
>> log.segment.delete.delay.ms = 60000
>> log.dirs = /data2/kafka-log
>> controlled.shutdown.enable = true
>> compression.type = producer
>> max.connections.per.ip.overrides =
>> log.message.timestamp.difference.max.ms = 9223372036854775807
>> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>> auto.leader.rebalance.enable = true
>> leader.imbalance.check.interval.seconds = 300
>> log.cleaner.min.cleanable.ratio = 0.5
>> replica.lag.time.max.ms = 10000
>> num.network.threads = 3
>> ssl.key.password = null
>> reserved.broker.max.id = 1000
>> metrics.num.samples = 2
>> socket.send.buffer.bytes = 102400
>> ssl.protocol = TLS
>> socket.receive.buffer.bytes = 102400
>> ssl.keystore.location = null
>> replica.fetch.min.bytes = 1
>> broker.rack = null
>> unclean.leader.election.enable = true
>> sasl.enabled.mechanisms = [GSSAPI]
>> group.min.session.timeout.ms = 6000
>> log.cleaner.io.buffer.load.factor = 0.9
>> offsets.retention.check.interval.ms = 600000
>> producer.purgatory.purge.interval.requests = 10000
>> metrics.sample.window.ms = 30000
>> broker.id = -1
>> offsets.topic.compression.codec = 0
>> log.retention.check.interval.ms = 300000
>> advertised.listeners = null
>> leader.imbalance.per.broker.percentage = 10
>> (kafka.server.KafkaConfig)
>> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
>> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
>> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
>> (kafka.server.KafkaServer)
>> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer
>> startup.
>> Prepare to shutdown (kafka.server.KafkaServer)
>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>> zookeeper server within timeout: 25000
>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>> scala:75)
>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>> at
>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>> le.scala:37)
>> at kafka.Kafka$.main(Kafka.scala:67)
>> at kafka.Kafka.main(Kafka.scala)
>> [2017-05-17 17:03:40,740] INFO shutting down
>> (kafka.server.KafkaServer)
>> [2017-05-17 17:03:40,747] INFO shut down completed
>> (kafka.server.KafkaServer)
>> [2017-05-17 17:03:40,748] FATAL Fatal error during
>> KafkaServerStartable
>> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
>> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
>> zookeeper server within timeout: 25000
>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
>> scala:75)
>> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
>> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
>> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
>> at
>> kafka.server.KafkaServerStartable.startup(KafkaServerStartab
>> le.scala:37)
>> at kafka.Kafka$.main(Kafka.scala:67)
>> at kafka.Kafka.main(Kafka.scala)
>> [2017-05-17 17:03:40,764] INFO shutting down
>> (kafka.server.KafkaServer)
>>
>> Manually starting Kafka via Ambari fixes this... However, Metron REST
>> still doesn't work:
>>
>> . ____ _ __ _ _
>> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
>> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
>> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
>> ' |____| .__|_| |_|_| |_\__, | / / / /
>> =========|_|==============|___/=/_/_/_/
>> :: Spring Boot :: (v1.4.1.RELEASE)
>>
>> 2017-05-17T17:03:52.406 ERROR
>> [org.springframework.boot.SpringApplication]
>> - Application startup failed
>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>> Error
>> creating bean with name 'kafkaServiceImpl' defined in URL
>> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
>> INF/classes!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
>> Unsatisfied dependency expressed through constructor parameter 0;
>> nested
>> exception is
>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>> expressed through field 'zkClient'; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error
>> creating
>> bean with name 'zkClient' defined in class path resource
>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>> instantiation
>> via factory method failed; nested exception is
>> org.springframework.beans.BeanInstantiationException: Failed to
>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>> threw exception; nested exception is
>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>> Unable to connect to zookeeper server within timeout: 10000
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.createArgumentArray(ConstructorResolver.java:749)
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.autowireConstructor(ConstructorResolver.java:189)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.autowireConstructor(AbstractAu
>> towireCapableBeanFactory.java:1148)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBeanInstance(AbstractAut
>> owireCapableBeanFactory.java:1051)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBe
>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.getBean(AbstractBeanFactory.java:197)
>> at
>> org.springframework.beans.factory.support.DefaultListableBea
>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
>> at
>> org.springframework.context.support.AbstractApplicationConte
>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
>> at
>> org.springframework.context.support.AbstractApplicationConte
>> xt.refresh(AbstractApplicationContext.java:541)
>> at
>> org.springframework.boot.context.embedded.EmbeddedWebApplica
>> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
>> at
>> org.springframework.boot.SpringApplication.refresh(SpringApp
>> lication.java:761)
>> at
>> org.springframework.boot.SpringApplication.refreshContext(Sp
>> ringApplication.java:371)
>> at
>> org.springframework.boot.SpringApplication.run(SpringApplica
>> tion.java:315)
>> at
>> org.springframework.boot.SpringApplication.run(SpringApplica
>> tion.java:1186)
>> at
>> org.springframework.boot.SpringApplication.run(SpringApplica
>> tion.java:1175)
>> at
>> org.apache.metron.rest.MetronRestApplication.main(MetronRest
>> Application.java:27)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.boot.loader.MainMethodRunner.run(MainMet
>> hodRunner.java:48)
>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>> java:87)
>> at org.springframework.boot.loader.Launcher.launch(Launcher.
>> java:50)
>> at org.springframework.boot.loader.PropertiesLauncher.main(
>> PropertiesLauncher.java:525)
>> Caused by:
>> org.springframework.beans.factory.UnsatisfiedDependencyException:
>> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
>> expressed through field 'zkClient'; nested exception is
>> org.springframework.beans.factory.BeanCreationException: Error
>> creating
>> bean with name 'zkClient' defined in class path resource
>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>> instantiation
>> via factory method failed; nested exception is
>> org.springframework.beans.BeanInstantiationException: Failed to
>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>> threw exception; nested exception is
>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>> Unable to connect to zookeeper server within timeout: 10000
>> at
>> org.springframework.beans.factory.annotation.AutowiredAnnota
>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>> utowiredAnnotationBeanPostProcessor.java:569)
>> at
>> org.springframework.beans.factory.annotation.InjectionMetada
>> ta.inject(InjectionMetadata.java:88)
>> at
>> org.springframework.beans.factory.annotation.AutowiredAnnota
>> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
>> otationBeanPostProcessor.java:349)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
>> tory.java:1219)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBe
>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.getBean(AbstractBeanFactory.java:197)
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>> ireCapableBeanFactory.java:1128)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBeanInstance(AbstractAut
>> owireCapableBeanFactory.java:1023)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBe
>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.getBean(AbstractBeanFactory.java:202)
>> at
>> org.springframework.beans.factory.config.DependencyDescripto
>> r.resolveCandidate(DependencyDescriptor.java:207)
>> at
>> org.springframework.beans.factory.support.DefaultListableBea
>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>> at
>> org.springframework.beans.factory.support.DefaultListableBea
>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.resolveAutowiredArgument(ConstructorResolver.java:835)
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.createArgumentArray(ConstructorResolver.java:741)
>> ... 27 more
>> Caused by: org.springframework.beans.factory.BeanCreationException:
>> Error
>> creating bean with name 'zkClient' defined in class path resource
>> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
>> instantiation
>> via factory method failed; nested exception is
>> org.springframework.beans.BeanInstantiationException: Failed to
>> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
>> threw exception; nested exception is
>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>> Unable to connect to zookeeper server within timeout: 10000
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
>> ireCapableBeanFactory.java:1128)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBeanInstance(AbstractAut
>> owireCapableBeanFactory.java:1023)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
>> at
>> org.springframework.beans.factory.support.AbstractAutowireCa
>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y$1.getObject(AbstractBeanFactory.java:306)
>> at
>> org.springframework.beans.factory.support.DefaultSingletonBe
>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.doGetBean(AbstractBeanFactory.java:302)
>> at
>> org.springframework.beans.factory.support.AbstractBeanFactor
>> y.getBean(AbstractBeanFactory.java:202)
>> at
>> org.springframework.beans.factory.config.DependencyDescripto
>> r.resolveCandidate(DependencyDescriptor.java:207)
>> at
>> org.springframework.beans.factory.support.DefaultListableBea
>> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
>> at
>> org.springframework.beans.factory.support.DefaultListableBea
>> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
>> at
>> org.springframework.beans.factory.annotation.AutowiredAnnota
>> tionBeanPostProcessor$AutowiredFieldElement.inject(A
>> utowiredAnnotationBeanPostProcessor.java:566)
>> ... 50 more
>> Caused by: org.springframework.beans.BeanInstantiationException:
>> Failed
>> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method
>> 'zkClient'
>> threw exception; nested exception is
>> org.I0Itec.zkclient.exception.ZkTimeoutException:
>> Unable to connect to zookeeper server within timeout: 10000
>> at
>> org.springframework.beans.factory.support.SimpleInstantiatio
>> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
>> at
>> org.springframework.beans.factory.support.ConstructorResolve
>> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
>> ... 62 more
>> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
>> connect to zookeeper server within timeout: 10000
>> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
>> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
>> at
>> org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
>> eperConfig.java:52)
>> at
>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
>> at
>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(<generated>)
>> at
>> org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
>> odProxy.java:228)
>> at
>> org.springframework.context.annotation.ConfigurationClassEnh
>> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
>> at
>> org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
>> ingCGLIB$$6e0c6742.zkClient(<generated>)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.springframework.beans.factory.support.SimpleInstantiatio
>> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
>> ... 63 more
>>
>> Any ideas on what might be going or why the timeout triggers?
>>
>>
>> On 2017-05-17 08:31, Laurens Vets wrote:
>>
>>> Hello list,
>>>
>>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>>> following changes in playbook.yml so that I'm left with 5 EC2
>>> instances instead of 10:
>>>
>>> <-- SNIP -->
>>> tasks:
>>> - include: tasks/create-keypair.yml
>>> - include: tasks/create-vpc.yml
>>> - include: tasks/create-open-inbound-security-group.yml
>>> - include: tasks/create-open-outbound-security-group.yml
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=sensors,ambari_master,ec2,monit
>>> # - include: tasks/create-hosts.yml host_count=4
>>> host_type=ambari_slave,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=pcap_server,monit,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>>> # - include: tasks/create-hosts.yml host_count=2
>>> host_type=ambari_slave,search,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,search,ec2
>>> - include: tasks/create-hosts.yml host_count=1
>>> host_type=ambari_slave,web,ec2
>>> <-- END SNIP -->
>>>
>>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>>> playbook.yml \
>>> --skip-tags="quick_dev,sensor-stubs" \
>>> --extra-vars="env=$ENV" \
>>> $EXTRA_ARGS"
>>>
>>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>>> --skip-tags="quick_dev,sensor-stubs" \
>>> --extra-vars="env=$ENV" \
>>> $EXTRA_ARGS -c paramiko"
>>> in run.sh otherwise ssh would time out.
>>>
>>> However, Ansible errors out with:
>>>
>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>>> Deploy cluster with Ambari;
>>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>>> ambari_config/tasks/main.yml:36
>>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>>> {"changed": false, "failed": true, "invocation": {"module_args":
>>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
>>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [],
>>> "name":
>>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>>> "name": "slaves"}], "required_configurations": [{"metron-env":
>>> {"es_hosts":
>>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>>> 8-94-45.us-west-2.compute.amazonaws.com",
>>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password":
>>> "root",
>>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>>> "storm_rest_addr":
>>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>>> "zeppelin_server_url":
>>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
>>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>>> "kibana_es_url":
>>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name":
>>> "HDP",
>>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
>>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>>> {"dfs.datanode.data.dir":
>>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>>> "yarn.nodemanager.resource.memory-mb": 4096,
>>> "yarn.timeline-service.leveldb-state-store.path":
>>> "/data1/hadoop/yarn/timeline",
>>> "yarn.timeline-service.leveldb-timeline-store.path":
>>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas":
>>> 1,
>>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>>> "zen_discovery_ping_unicast_hosts": "[
>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>>> exception occurred: Could not create cluster: request code 400,
>>> request message {\n \"status\" : 400,\n \"message\" :
>>> \"Invalid Cluster Creation Template:
>>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>>> 'host_count' value\"\n}"}
>>>
>>> PLAY RECAP
>>> ************************************************************
>>> *********
>>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
>>> unreachable=0 failed=0
>>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23
>>> changed=8
>>> unreachable=0 failed=0
>>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58
>>> changed=31
>>> unreachable=0 failed=0
>>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49
>>> changed=27
>>> unreachable=0 failed=0
>>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54
>>> changed=28
>>> unreachable=0 failed=1
>>> localhost : ok=29 changed=12 unreachable=0
>>> failed=0
>>>
>>> Any idea what might be going on?
>>>
>>> (Don't worry about the URLs, the instances have been terminated)
>>>
>>
Re: Trying to spin up Metron in EC2: Failed
Posted by Ryan Merriman <me...@gmail.com>.
That happens when you don't have the zookeeper url configured correctly.
Can you check the contents of the /etc/sysconfig/metron file on the Metron
host?
On Wed, May 17, 2017 at 1:36 PM, Laurens Vets <la...@daemon.be> wrote:
> For testing purposes, I decided to spin up the default Metron AWS config.
> This resulted in a hang from ansible here:
>
> TASK [librdkafka : include] ******************************
> **********************
> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/main.yml:18
> included: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/dependencies.yml for ec2-34-210-194-189.us-west-2.c
> ompute.amazonaws.com
>
> TASK [librdkafka : Install prerequisites] ******************************
> ********
> task path: /home/laurens/SAPSource/metron/metron-deployment/roles/
> librdkafka/tasks/dependencies.yml:18
> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH CONNECTION
> FOR USER: centos on PORT 22 TO ec2-34-210-194-189.us-west-2.c
> ompute.amazonaws.com
> /usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
> FutureWarning: CTR mode needs counter parameter, not IV
> self._cipher = factory.new(key, *args, **kwargs)
> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 &&
> mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
> )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508
> )" )
> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT /tmp/tmpwnH61y
> TO /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum
> <ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
> 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh;
> LANG=en_CA.UTF-8 LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 /usr/bin/python
> -tt /home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum;
> rm -rf "/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/"
> > /dev/null 2>&1'"'"''
>
> Looking in the machine logs, I see the following for Kafka and Metron REST:
>
> Kafka:
> [2017-05-17 17:03:14,831] INFO KafkaConfig values:
> advertised.host.name = null
> metric.reporters = []
> quota.producer.default = 9223372036854775807
> offsets.topic.num.partitions = 50
> log.flush.interval.messages = 9223372036854775807
> auto.create.topics.enable = true
> controller.socket.timeout.ms = 30000
> log.flush.interval.ms = null
> principal.builder.class = class org.apache.kafka.common.securi
> ty.auth.DefaultPrincipalBuilder
> replica.socket.receive.buffer.bytes = 65536
> min.insync.replicas = 1
> replica.fetch.wait.max.ms = 500
> num.recovery.threads.per.data.dir = 1
> ssl.keystore.type = JKS
> sasl.mechanism.inter.broker.protocol = GSSAPI
> default.replication.factor = 1
> ssl.truststore.password = null
> log.preallocate = false
> sasl.kerberos.principal.to.local.rules = [DEFAULT]
> fetch.purgatory.purge.interval.requests = 10000
> ssl.endpoint.identification.algorithm = null
> replica.socket.timeout.ms = 30000
> message.max.bytes = 1000000
> num.io.threads = 8
> offsets.commit.required.acks = -1
> log.flush.offset.checkpoint.interval.ms = 60000
> delete.topic.enable = false
> quota.window.size.seconds = 1
> ssl.truststore.type = JKS
> offsets.commit.timeout.ms = 5000
> quota.window.num = 11
> zookeeper.connect = ec2-34-223-200-113.us-west-2.c
> ompute.amazonaws.com:2181
> authorizer.class.name =
> num.replica.fetchers = 1
> log.retention.ms = null
> log.roll.jitter.hours = 0
> log.cleaner.enable = true
> offsets.load.buffer.size = 5242880
> log.cleaner.delete.retention.ms = 86400000
> ssl.client.auth = none
> controlled.shutdown.max.retries = 3
> queued.max.requests = 500
> offsets.topic.replication.factor = 3
> log.cleaner.threads = 1
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> socket.request.max.bytes = 104857600
> ssl.trustmanager.algorithm = PKIX
> zookeeper.session.timeout.ms = 30000
> log.retention.bytes = -1
> log.message.timestamp.type = CreateTime
> sasl.kerberos.min.time.before.relogin = 60000
> zookeeper.set.acl = false
> connections.max.idle.ms = 600000
> offsets.retention.minutes = 86400000
> replica.fetch.backoff.ms = 1000
> inter.broker.protocol.version = 0.10.0-IV1
> log.retention.hours = 168
> num.partitions = 1
> broker.id.generation.enable = true
> listeners = PLAINTEXT://ec2-34-209-53-166.
> us-west-2.compute.amazonaws.com:6667
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> log.roll.ms = null
> log.flush.scheduler.interval.ms = 9223372036854775807
> ssl.cipher.suites = null
> log.index.size.max.bytes = 10485760
> ssl.keymanager.algorithm = SunX509
> security.inter.broker.protocol = PLAINTEXT
> replica.fetch.max.bytes = 1048576
> advertised.port = null
> log.cleaner.dedupe.buffer.size = 134217728
> replica.high.watermark.checkpoint.interval.ms = 5000
> log.cleaner.io.buffer.size = 524288
> sasl.kerberos.ticket.renew.window.factor = 0.8
> zookeeper.connection.timeout.ms = 25000
> controlled.shutdown.retry.backoff.ms = 5000
> log.roll.hours = 168
> log.cleanup.policy = delete
> host.name =
> log.roll.jitter.ms = null
> max.connections.per.ip = 2147483647
> offsets.topic.segment.bytes = 104857600
> background.threads = 10
> quota.consumer.default = 9223372036854775807
> request.timeout.ms = 30000
> log.message.format.version = 0.10.0-IV1
> log.index.interval.bytes = 4096
> log.dir = /tmp/kafka-logs
> log.segment.bytes = 1073741824
> log.cleaner.backoff.ms = 15000
> offset.metadata.max.bytes = 4096
> ssl.truststore.location = null
> group.max.session.timeout.ms = 300000
> ssl.keystore.password = null
> zookeeper.sync.time.ms = 2000
> port = 6667
> log.retention.minutes = null
> log.segment.delete.delay.ms = 60000
> log.dirs = /data2/kafka-log
> controlled.shutdown.enable = true
> compression.type = producer
> max.connections.per.ip.overrides =
> log.message.timestamp.difference.max.ms = 9223372036854775807
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
> auto.leader.rebalance.enable = true
> leader.imbalance.check.interval.seconds = 300
> log.cleaner.min.cleanable.ratio = 0.5
> replica.lag.time.max.ms = 10000
> num.network.threads = 3
> ssl.key.password = null
> reserved.broker.max.id = 1000
> metrics.num.samples = 2
> socket.send.buffer.bytes = 102400
> ssl.protocol = TLS
> socket.receive.buffer.bytes = 102400
> ssl.keystore.location = null
> replica.fetch.min.bytes = 1
> broker.rack = null
> unclean.leader.election.enable = true
> sasl.enabled.mechanisms = [GSSAPI]
> group.min.session.timeout.ms = 6000
> log.cleaner.io.buffer.load.factor = 0.9
> offsets.retention.check.interval.ms = 600000
> producer.purgatory.purge.interval.requests = 10000
> metrics.sample.window.ms = 30000
> broker.id = -1
> offsets.topic.compression.codec = 0
> log.retention.check.interval.ms = 300000
> advertised.listeners = null
> leader.imbalance.per.broker.percentage = 10
> (kafka.server.KafkaConfig)
> [2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
> [2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
> ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
> (kafka.server.KafkaServer)
> [2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup.
> Prepare to shutdown (kafka.server.KafkaServer)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
> zookeeper server within timeout: 25000
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
> scala:75)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab
> le.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer)
> [2017-05-17 17:03:40,747] INFO shut down completed
> (kafka.server.KafkaServer)
> [2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
> zookeeper server within timeout: 25000
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.
> scala:75)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
> at kafka.server.KafkaServerStartable.startup(KafkaServerStartab
> le.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> [2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer)
>
> Manually starting Kafka via Ambari fixes this... However, Metron REST
> still doesn't work:
>
> . ____ _ __ _ _
> /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
> ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
> \\/ ___)| |_)| | | | | || (_| | ) ) ) )
> ' |____| .__|_| |_|_| |_\__, | / / / /
> =========|_|==============|___/=/_/_/_/
> :: Spring Boot :: (v1.4.1.RELEASE)
>
> 2017-05-17T17:03:52.406 ERROR [org.springframework.boot.SpringApplication]
> - Application startup failed
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error
> creating bean with name 'kafkaServiceImpl' defined in URL
> [jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-
> INF/classes!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
> Unsatisfied dependency expressed through constructor parameter 0; nested
> exception is org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
> expressed through field 'zkClient'; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'zkClient' defined in class path resource
> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation
> via factory method failed; nested exception is
> org.springframework.beans.BeanInstantiationException: Failed to
> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
> threw exception; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException:
> Unable to connect to zookeeper server within timeout: 10000
> at org.springframework.beans.factory.support.ConstructorResolve
> r.createArgumentArray(ConstructorResolver.java:749)
> at org.springframework.beans.factory.support.ConstructorResolve
> r.autowireConstructor(ConstructorResolver.java:189)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.autowireConstructor(AbstractAu
> towireCapableBeanFactory.java:1148)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBeanInstance(AbstractAut
> owireCapableBeanFactory.java:1051)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBe
> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.getBean(AbstractBeanFactory.java:197)
> at org.springframework.beans.factory.support.DefaultListableBea
> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
> at org.springframework.context.support.AbstractApplicationConte
> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
> at org.springframework.context.support.AbstractApplicationConte
> xt.refresh(AbstractApplicationContext.java:541)
> at org.springframework.boot.context.embedded.EmbeddedWebApplica
> tionContext.refresh(EmbeddedWebApplicationContext.java:122)
> at org.springframework.boot.SpringApplication.refresh(SpringApp
> lication.java:761)
> at org.springframework.boot.SpringApplication.refreshContext(Sp
> ringApplication.java:371)
> at org.springframework.boot.SpringApplication.run(SpringApplica
> tion.java:315)
> at org.springframework.boot.SpringApplication.run(SpringApplica
> tion.java:1186)
> at org.springframework.boot.SpringApplication.run(SpringApplica
> tion.java:1175)
> at org.apache.metron.rest.MetronRestApplication.main(MetronRest
> Application.java:27)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.boot.loader.MainMethodRunner.run(MainMet
> hodRunner.java:48)
> at org.springframework.boot.loader.Launcher.launch(Launcher.
> java:87)
> at org.springframework.boot.loader.Launcher.launch(Launcher.
> java:50)
> at org.springframework.boot.loader.PropertiesLauncher.main(
> PropertiesLauncher.java:525)
> Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException:
> Error creating bean with name 'kafkaConfig': Unsatisfied dependency
> expressed through field 'zkClient'; nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'zkClient' defined in class path resource
> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation
> via factory method failed; nested exception is
> org.springframework.beans.BeanInstantiationException: Failed to
> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
> threw exception; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException:
> Unable to connect to zookeeper server within timeout: 10000
> at org.springframework.beans.factory.annotation.AutowiredAnnota
> tionBeanPostProcessor$AutowiredFieldElement.inject(A
> utowiredAnnotationBeanPostProcessor.java:569)
> at org.springframework.beans.factory.annotation.InjectionMetada
> ta.inject(InjectionMetadata.java:88)
> at org.springframework.beans.factory.annotation.AutowiredAnnota
> tionBeanPostProcessor.postProcessPropertyValues(AutowiredAnn
> otationBeanPostProcessor.java:349)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.populateBean(AbstractAutowireCapableBeanFac
> tory.java:1219)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBe
> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.getBean(AbstractBeanFactory.java:197)
> at org.springframework.beans.factory.support.ConstructorResolve
> r.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
> ireCapableBeanFactory.java:1128)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBeanInstance(AbstractAut
> owireCapableBeanFactory.java:1023)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBe
> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.getBean(AbstractBeanFactory.java:202)
> at org.springframework.beans.factory.config.DependencyDescripto
> r.resolveCandidate(DependencyDescriptor.java:207)
> at org.springframework.beans.factory.support.DefaultListableBea
> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
> at org.springframework.beans.factory.support.DefaultListableBea
> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
> at org.springframework.beans.factory.support.ConstructorResolve
> r.resolveAutowiredArgument(ConstructorResolver.java:835)
> at org.springframework.beans.factory.support.ConstructorResolve
> r.createArgumentArray(ConstructorResolver.java:741)
> ... 27 more
> Caused by: org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'zkClient' defined in class path resource
> [org/apache/metron/rest/config/ZookeeperConfig.class]: Bean instantiation
> via factory method failed; nested exception is
> org.springframework.beans.BeanInstantiationException: Failed to
> instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
> threw exception; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException:
> Unable to connect to zookeeper server within timeout: 10000
> at org.springframework.beans.factory.support.ConstructorResolve
> r.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.instantiateUsingFactoryMethod(AbstractAutow
> ireCapableBeanFactory.java:1128)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBeanInstance(AbstractAut
> owireCapableBeanFactory.java:1023)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
> at org.springframework.beans.factory.support.AbstractAutowireCa
> pableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y$1.getObject(AbstractBeanFactory.java:306)
> at org.springframework.beans.factory.support.DefaultSingletonBe
> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.doGetBean(AbstractBeanFactory.java:302)
> at org.springframework.beans.factory.support.AbstractBeanFactor
> y.getBean(AbstractBeanFactory.java:202)
> at org.springframework.beans.factory.config.DependencyDescripto
> r.resolveCandidate(DependencyDescriptor.java:207)
> at org.springframework.beans.factory.support.DefaultListableBea
> nFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
> at org.springframework.beans.factory.support.DefaultListableBea
> nFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
> at org.springframework.beans.factory.annotation.AutowiredAnnota
> tionBeanPostProcessor$AutowiredFieldElement.inject(A
> utowiredAnnotationBeanPostProcessor.java:566)
> ... 50 more
> Caused by: org.springframework.beans.BeanInstantiationException: Failed
> to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
> threw exception; nested exception is org.I0Itec.zkclient.exception.ZkTimeoutException:
> Unable to connect to zookeeper server within timeout: 10000
> at org.springframework.beans.factory.support.SimpleInstantiatio
> nStrategy.instantiate(SimpleInstantiationStrategy.java:189)
> at org.springframework.beans.factory.support.ConstructorResolve
> r.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
> ... 62 more
> Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
> connect to zookeeper server within timeout: 10000
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
> at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
> at org.apache.metron.rest.config.ZookeeperConfig.zkClient(Zooke
> eperConfig.java:52)
> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
> ingCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
> ingCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invokeSuper(Meth
> odProxy.java:228)
> at org.springframework.context.annotation.ConfigurationClassEnh
> ancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
> at org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpr
> ingCGLIB$$6e0c6742.zkClient(<generated>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.beans.factory.support.SimpleInstantiatio
> nStrategy.instantiate(SimpleInstantiationStrategy.java:162)
> ... 63 more
>
> Any ideas on what might be going or why the timeout triggers?
>
>
> On 2017-05-17 08:31, Laurens Vets wrote:
>
>> Hello list,
>>
>> I'm trying to spin up Metron in EC2 for testing purposes. I made the
>> following changes in playbook.yml so that I'm left with 5 EC2
>> instances instead of 10:
>>
>> <-- SNIP -->
>> tasks:
>> - include: tasks/create-keypair.yml
>> - include: tasks/create-vpc.yml
>> - include: tasks/create-open-inbound-security-group.yml
>> - include: tasks/create-open-outbound-security-group.yml
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=sensors,ambari_master,ec2,monit
>> # - include: tasks/create-hosts.yml host_count=4
>> host_type=ambari_slave,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=pcap_server,monit,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
>> # - include: tasks/create-hosts.yml host_count=2
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,search,ec2
>> - include: tasks/create-hosts.yml host_count=1
>> host_type=ambari_slave,web,ec2
>> <-- END SNIP -->
>>
>> I also had to change the line "ansible-playbook -vvv -i ec2.py
>> playbook.yml \
>> --skip-tags="quick_dev,sensor-stubs" \
>> --extra-vars="env=$ENV" \
>> $EXTRA_ARGS"
>>
>> to "ansible-playbook -vvv -i ec2.py playbook.yml \
>> --skip-tags="quick_dev,sensor-stubs" \
>> --extra-vars="env=$ENV" \
>> $EXTRA_ARGS -c paramiko"
>> in run.sh otherwise ssh would time out.
>>
>> However, Ansible errors out with:
>>
>> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
>> Deploy cluster with Ambari;
>> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
>> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
>> /home/laurens/SAPSource/metron/metron-deployment/roles/
>> ambari_config/tasks/main.yml:36
>> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
>> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
>> {"changed": false, "failed": true, "invocation": {"module_args":
>> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
>> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
>> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
>> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
>> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
>> "configuration": [], "name": "master_1"}, {"cardinality": 1,
>> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
>> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
>> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
>> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
>> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
>> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
>> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>> "name": "metron"}, {"cardinality": 1, "components": [{"name":
>> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name":
>> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
>> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
>> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
>> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
>> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
>> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
>> "name": "slaves"}], "required_configurations": [{"metron-env":
>> {"es_hosts":
>> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-20
>> 8-94-45.us-west-2.compute.amazonaws.com",
>> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root",
>> "metron_jdbc_platform": "h2", "metron_jdbc_url":
>> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
>> "storm_rest_addr":
>> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
>> "zeppelin_server_url":
>> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
>> {"kibana_default_application": "dashboard/Metron-Dashboard",
>> "kibana_es_url":
>> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
>> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
>> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP",
>> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
>> "present", "configurations": [{"zoo.cfg": {"dataDir":
>> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
>> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
>> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
>> {"dfs.datanode.data.dir":
>> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
>> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
>> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
>> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
>> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
>> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
>> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
>> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
>> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
>> "yarn.nodemanager.resource.memory-mb": 4096,
>> "yarn.timeline-service.leveldb-state-store.path":
>> "/data1/hadoop/yarn/timeline",
>> "yarn.timeline-service.leveldb-timeline-store.path":
>> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
>> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
>> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
>> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
>> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
>> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1,
>> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
>> "zen_discovery_ping_unicast_hosts": "[
>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
>> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
>> "admin", "port": 8080, "username": "admin", "wait_for_complete":
>> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
>> exception occurred: Could not create cluster: request code 400,
>> request message {\n \"status\" : 400,\n \"message\" :
>> \"Invalid Cluster Creation Template:
>> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
>> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
>> 'host_count' value\"\n}"}
>>
>> PLAY RECAP ************************************************************
>> *********
>> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
>> unreachable=0 failed=0
>> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8
>> unreachable=0 failed=0
>> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31
>> unreachable=0 failed=0
>> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27
>> unreachable=0 failed=0
>> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28
>> unreachable=0 failed=1
>> localhost : ok=29 changed=12 unreachable=0
>> failed=0
>>
>> Any idea what might be going on?
>>
>> (Don't worry about the URLs, the instances have been terminated)
>>
>
Re: Trying to spin up Metron in EC2: Failed
Posted by Laurens Vets <la...@daemon.be>.
For testing purposes, I decided to spin up the default Metron AWS
config. This resulted in a hang from ansible here:
TASK [librdkafka : include]
****************************************************
task path:
/home/laurens/SAPSource/metron/metron-deployment/roles/librdkafka/tasks/main.yml:18
included:
/home/laurens/SAPSource/metron/metron-deployment/roles/librdkafka/tasks/dependencies.yml
for ec2-34-210-194-189.us-west-2.compute.amazonaws.com
TASK [librdkafka : Install prerequisites]
**************************************
task path:
/home/laurens/SAPSource/metron/metron-deployment/roles/librdkafka/tasks/dependencies.yml:18
<ec2-34-210-194-189.us-west-2.compute.amazonaws.com> ESTABLISH
CONNECTION FOR USER: centos on PORT 22 TO
ec2-34-210-194-189.us-west-2.compute.amazonaws.com
/usr/lib/python2.7/dist-packages/Crypto/Cipher/blockalgo.py:141:
FutureWarning: CTR mode needs counter parameter, not IV
self._cipher = factory.new(key, *args, **kwargs)
<ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC ( umask 22 &&
mkdir -p "$( echo
$HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508 )" && echo
"$( echo $HOME/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508 )"
)
<ec2-34-210-194-189.us-west-2.compute.amazonaws.com> PUT /tmp/tmpwnH61y
TO
/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum
<ec2-34-210-194-189.us-west-2.compute.amazonaws.com> EXEC /bin/sh -c
'sudo -H -S -n -u root /bin/sh -c '"'"'echo
BECOME-SUCCESS-rmswjjyhfdywqvwtvqwcmbsqpsbohvxh; LANG=en_CA.UTF-8
LC_ALL=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8 /usr/bin/python -tt
/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/yum;
rm -rf
"/home/centos/.ansible/tmp/ansible-tmp-1495041091.74-92163853889508/" >
/dev/null 2>&1'"'"''
Looking in the machine logs, I see the following for Kafka and Metron
REST:
Kafka:
[2017-05-17 17:03:14,831] INFO KafkaConfig values:
advertised.host.name = null
metric.reporters = []
quota.producer.default = 9223372036854775807
offsets.topic.num.partitions = 50
log.flush.interval.messages = 9223372036854775807
auto.create.topics.enable = true
controller.socket.timeout.ms = 30000
log.flush.interval.ms = null
principal.builder.class = class
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
replica.socket.receive.buffer.bytes = 65536
min.insync.replicas = 1
replica.fetch.wait.max.ms = 500
num.recovery.threads.per.data.dir = 1
ssl.keystore.type = JKS
sasl.mechanism.inter.broker.protocol = GSSAPI
default.replication.factor = 1
ssl.truststore.password = null
log.preallocate = false
sasl.kerberos.principal.to.local.rules = [DEFAULT]
fetch.purgatory.purge.interval.requests = 10000
ssl.endpoint.identification.algorithm = null
replica.socket.timeout.ms = 30000
message.max.bytes = 1000000
num.io.threads = 8
offsets.commit.required.acks = -1
log.flush.offset.checkpoint.interval.ms = 60000
delete.topic.enable = false
quota.window.size.seconds = 1
ssl.truststore.type = JKS
offsets.commit.timeout.ms = 5000
quota.window.num = 11
zookeeper.connect =
ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
authorizer.class.name =
num.replica.fetchers = 1
log.retention.ms = null
log.roll.jitter.hours = 0
log.cleaner.enable = true
offsets.load.buffer.size = 5242880
log.cleaner.delete.retention.ms = 86400000
ssl.client.auth = none
controlled.shutdown.max.retries = 3
queued.max.requests = 500
offsets.topic.replication.factor = 3
log.cleaner.threads = 1
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
socket.request.max.bytes = 104857600
ssl.trustmanager.algorithm = PKIX
zookeeper.session.timeout.ms = 30000
log.retention.bytes = -1
log.message.timestamp.type = CreateTime
sasl.kerberos.min.time.before.relogin = 60000
zookeeper.set.acl = false
connections.max.idle.ms = 600000
offsets.retention.minutes = 86400000
replica.fetch.backoff.ms = 1000
inter.broker.protocol.version = 0.10.0-IV1
log.retention.hours = 168
num.partitions = 1
broker.id.generation.enable = true
listeners =
PLAINTEXT://ec2-34-209-53-166.us-west-2.compute.amazonaws.com:6667
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
log.roll.ms = null
log.flush.scheduler.interval.ms = 9223372036854775807
ssl.cipher.suites = null
log.index.size.max.bytes = 10485760
ssl.keymanager.algorithm = SunX509
security.inter.broker.protocol = PLAINTEXT
replica.fetch.max.bytes = 1048576
advertised.port = null
log.cleaner.dedupe.buffer.size = 134217728
replica.high.watermark.checkpoint.interval.ms = 5000
log.cleaner.io.buffer.size = 524288
sasl.kerberos.ticket.renew.window.factor = 0.8
zookeeper.connection.timeout.ms = 25000
controlled.shutdown.retry.backoff.ms = 5000
log.roll.hours = 168
log.cleanup.policy = delete
host.name =
log.roll.jitter.ms = null
max.connections.per.ip = 2147483647
offsets.topic.segment.bytes = 104857600
background.threads = 10
quota.consumer.default = 9223372036854775807
request.timeout.ms = 30000
log.message.format.version = 0.10.0-IV1
log.index.interval.bytes = 4096
log.dir = /tmp/kafka-logs
log.segment.bytes = 1073741824
log.cleaner.backoff.ms = 15000
offset.metadata.max.bytes = 4096
ssl.truststore.location = null
group.max.session.timeout.ms = 300000
ssl.keystore.password = null
zookeeper.sync.time.ms = 2000
port = 6667
log.retention.minutes = null
log.segment.delete.delay.ms = 60000
log.dirs = /data2/kafka-log
controlled.shutdown.enable = true
compression.type = producer
max.connections.per.ip.overrides =
log.message.timestamp.difference.max.ms = 9223372036854775807
sasl.kerberos.kinit.cmd = /usr/bin/kinit
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
auto.leader.rebalance.enable = true
leader.imbalance.check.interval.seconds = 300
log.cleaner.min.cleanable.ratio = 0.5
replica.lag.time.max.ms = 10000
num.network.threads = 3
ssl.key.password = null
reserved.broker.max.id = 1000
metrics.num.samples = 2
socket.send.buffer.bytes = 102400
ssl.protocol = TLS
socket.receive.buffer.bytes = 102400
ssl.keystore.location = null
replica.fetch.min.bytes = 1
broker.rack = null
unclean.leader.election.enable = true
sasl.enabled.mechanisms = [GSSAPI]
group.min.session.timeout.ms = 6000
log.cleaner.io.buffer.load.factor = 0.9
offsets.retention.check.interval.ms = 600000
producer.purgatory.purge.interval.requests = 10000
metrics.sample.window.ms = 30000
broker.id = -1
offsets.topic.compression.codec = 0
log.retention.check.interval.ms = 300000
advertised.listeners = null
leader.imbalance.per.broker.percentage = 10
(kafka.server.KafkaConfig)
[2017-05-17 17:03:14,990] INFO starting (kafka.server.KafkaServer)
[2017-05-17 17:03:14,998] INFO Connecting to zookeeper on
ec2-34-223-200-113.us-west-2.compute.amazonaws.com:2181
(kafka.server.KafkaServer)
[2017-05-17 17:03:40,740] FATAL Fatal error during KafkaServer startup.
Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 25000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
at
kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-17 17:03:40,740] INFO shutting down (kafka.server.KafkaServer)
[2017-05-17 17:03:40,747] INFO shut down completed
(kafka.server.KafkaServer)
[2017-05-17 17:03:40,748] FATAL Fatal error during KafkaServerStartable
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 25000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
at
kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:294)
at kafka.server.KafkaServer.startup(KafkaServer.scala:180)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2017-05-17 17:03:40,764] INFO shutting down (kafka.server.KafkaServer)
Manually starting Kafka via Ambari fixes this... However, Metron REST
still doesn't work:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.4.1.RELEASE)
2017-05-17T17:03:52.406 ERROR
[org.springframework.boot.SpringApplication] - Application startup
failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name 'kafkaServiceImpl' defined in URL
[jar:file:/usr/metron/0.4.0/lib/metron-rest-0.4.0.jar!/BOOT-INF/classes!/org/apache/metron/rest/service/impl/KafkaServiceImpl.class]:
Unsatisfied dependency expressed through constructor parameter 0; nested
exception is
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name 'kafkaConfig': Unsatisfied dependency expressed
through field 'zkClient'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'zkClient' defined in class path resource
[org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
threw exception; nested exception is
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 10000
at
org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:749)
at
org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1148)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1051)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:751)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:861)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
at
org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at
org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761)
at
org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:371)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1186)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1175)
at
org.apache.metron.rest.MetronRestApplication.main(MetronRestApplication.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at
org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at
org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at
org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:525)
Caused by:
org.springframework.beans.factory.UnsatisfiedDependencyException: Error
creating bean with name 'kafkaConfig': Unsatisfied dependency expressed
through field 'zkClient'; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating
bean with name 'zkClient' defined in class path resource
[org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
threw exception; nested exception is
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 10000
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:569)
at
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:349)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1219)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1023)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at
org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:207)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
at
org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
at
org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
... 27 more
Caused by: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'zkClient' defined in class path resource
[org/apache/metron/rest/config/ZookeeperConfig.class]: Bean
instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to
instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
threw exception; nested exception is
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 10000
at
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1023)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at
org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:207)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1128)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1056)
at
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:566)
... 50 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed
to instantiate [org.I0Itec.zkclient.ZkClient]: Factory method 'zkClient'
threw exception; nested exception is
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 10000
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 62 more
Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to
connect to zookeeper server within timeout: 10000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:97)
at
org.apache.metron.rest.config.ZookeeperConfig.zkClient(ZookeeperConfig.java:52)
at
org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$6e0c6742.CGLIB$zkClient$1(<generated>)
at
org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$6e0c6742$$FastClassBySpringCGLIB$$3219d861.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at
org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
at
org.apache.metron.rest.config.ZookeeperConfig$$EnhancerBySpringCGLIB$$6e0c6742.zkClient(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
... 63 more
Any ideas on what might be going or why the timeout triggers?
On 2017-05-17 08:31, Laurens Vets wrote:
> Hello list,
>
> I'm trying to spin up Metron in EC2 for testing purposes. I made the
> following changes in playbook.yml so that I'm left with 5 EC2
> instances instead of 10:
>
> <-- SNIP -->
> tasks:
> - include: tasks/create-keypair.yml
> - include: tasks/create-vpc.yml
> - include: tasks/create-open-inbound-security-group.yml
> - include: tasks/create-open-outbound-security-group.yml
> - include: tasks/create-hosts.yml host_count=1
> host_type=sensors,ambari_master,ec2,monit
> # - include: tasks/create-hosts.yml host_count=4
> host_type=ambari_slave,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=pcap_server,monit,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,enrichment,metron,ec2,zeppelin
> # - include: tasks/create-hosts.yml host_count=2
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,search,ec2
> - include: tasks/create-hosts.yml host_count=1
> host_type=ambari_slave,web,ec2
> <-- END SNIP -->
>
> I also had to change the line "ansible-playbook -vvv -i ec2.py
> playbook.yml \
> --skip-tags="quick_dev,sensor-stubs" \
> --extra-vars="env=$ENV" \
> $EXTRA_ARGS"
>
> to "ansible-playbook -vvv -i ec2.py playbook.yml \
> --skip-tags="quick_dev,sensor-stubs" \
> --extra-vars="env=$ENV" \
> $EXTRA_ARGS -c paramiko"
> in run.sh otherwise ssh would time out.
>
> However, Ansible errors out with:
>
> 2017-05-16 18:02:50,615 p=5362 u=laurens | TASK [ambari_config :
> Deploy cluster with Ambari;
> http://ec2-35-164-26-255.us-west-2.compute.amazonaws.com:8080] ***
> 2017-05-16 18:02:50,615 p=5362 u=laurens | task path:
> /home/laurens/SAPSource/metron/metron-deployment/roles/ambari_config/tasks/main.yml:36
> 2017-05-16 18:02:52,660 p=5362 u=laurens | fatal:
> [ec2-35-164-26-255.us-west-2.compute.amazonaws.com]: FAILED! =>
> {"changed": false, "failed": true, "invocation": {"module_args":
> {"blueprint_name": "metron_blueprint", "blueprint_var": {"groups":
> [{"cardinality": 1, "components": [{"name": "NAMENODE"}, {"name":
> "SECONDARY_NAMENODE"}, {"name": "RESOURCEMANAGER"}, {"name":
> "HISTORYSERVER"}, {"name": "HDFS_CLIENT"}, {"name": "YARN_CLIENT"},
> {"name": "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name": "ES_SLAVE"}],
> "configuration": [], "name": "master_1"}, {"cardinality": 1,
> "components": [{"name": "ZOOKEEPER_SERVER"}, {"name": "NIMBUS"},
> {"name": "STORM_UI_SERVER"}, {"name": "DRPC_SERVER"}, {"name":
> "SPARK_JOBHISTORYSERVER"}, {"name": "HBASE_MASTER"}, {"name":
> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}, {"name":
> "APP_TIMELINE_SERVER"}, {"name": "ES_SLAVE"}], "configuration": [],
> "name": "master_2"}, {"cardinality": 1, "components": [{"name":
> "METRON_INDEXING"}, {"name": "METRON_ENRICHMENT_MASTER"}, {"name":
> "METRON_PARSERS"}, {"name": "METRON_REST"}, {"name": "DATANODE"},
> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
> "name": "metron"}, {"cardinality": 1, "components": [{"name":
> "KIBANA_MASTER"}, {"name": "ES_MASTER"}], "configuration": [], "name":
> "web"}, {"cardinality": "1+", "components": [{"name": "DATANODE"},
> {"name": "NODEMANAGER"}, {"name": "SUPERVISOR"}, {"name":
> "KAFKA_BROKER"}, {"name": "HBASE_REGIONSERVER"}, {"name":
> "HDFS_CLIENT"}, {"name": "YARN_CLIENT"}, {"name":
> "MAPREDUCE2_CLIENT"}, {"name": "SPARK_CLIENT"}, {"name":
> "ZOOKEEPER_CLIENT"}, {"name": "HBASE_CLIENT"}], "configuration": [],
> "name": "slaves"}], "required_configurations": [{"metron-env":
> {"es_hosts":
> "ec2-35-164-136-88.us-west-2.compute.amazonaws.com,ec2-34-208-94-45.us-west-2.compute.amazonaws.com",
> "metron_jdbc_driver": "org.h2.Driver", "metron_jdbc_password": "root",
> "metron_jdbc_platform": "h2", "metron_jdbc_url":
> "jdbc:h2:file:~/metrondb", "metron_jdbc_username": "root",
> "storm_rest_addr":
> "ec2-34-208-94-45.us-west-2.compute.amazonaws.com:8744",
> "zeppelin_server_url":
> "ec2-34-210-151-29.us-west-2.compute.amazonaws.com"}}, {"kibana-env":
> {"kibana_default_application": "dashboard/Metron-Dashboard",
> "kibana_es_url":
> "http://ec2-35-164-136-88.us-west-2.compute.amazonaws.com:9200",
> "kibana_log_dir": "/var/log/kibana", "kibana_pid_dir":
> "/var/run/kibana", "kibana_server_port": 5000}}], "stack_name": "HDP",
> "stack_version": "2.5"}, "cluster_name": "metron", "cluster_state":
> "present", "configurations": [{"zoo.cfg": {"dataDir":
> "/data1/hadoop/zookeeper"}}, {"hadoop-env": {"dtnode_heapsize": 1024,
> "namenode_heapsize": 2048}}, {"hbase-env": {"hbase_master_heapsize":
> 1024, "hbase_regionserver_heapsize": 1024}}, {"hdfs-site":
> {"dfs.datanode.data.dir":
> "/data1/hadoop/hdfs/data,/data2/hadoop/hdfs/data",
> "dfs.journalnode.edits.dir": "/data1/hadoop/hdfs/journalnode",
> "dfs.namenode.checkpoint.dir": "/data1/hadoop/hdfs/namesecondary",
> "dfs.namenode.name.dir": "/data1/hadoop/hdfs/namenode"}},
> {"mapred-site": {"mapreduce.jobhistory.recovery.store.leveldb.path":
> "/data1/hadoop/mapreduce/jhs", "mapreduce.map.memory.mb": 1229,
> "mapreduce.reduce.memory.mb": 1229}}, {"yarn-site":
> {"yarn.nodemanager.local-dirs": "/data1/hadoop/yarn/local",
> "yarn.nodemanager.log-dirs": "/data1/hadoop/yarn/log",
> "yarn.nodemanager.resource.memory-mb": 4096,
> "yarn.timeline-service.leveldb-state-store.path":
> "/data1/hadoop/yarn/timeline",
> "yarn.timeline-service.leveldb-timeline-store.path":
> "/data1/hadoop/yarn/timeline"}}, {"storm-site": {"storm.local.dir":
> "/data1/hadoop/storm", "supervisor.slots.ports": "[6700, 6701, 6702,
> 6703]", "topology.classpath": "/etc/hbase/conf:/etc/hadoop/conf"}},
> {"kafka-broker": {"log.dirs": "/data2/kafka-log"}}, {"metron-env":
> {"parsers": "bro,snort,yaf"}}, {"elastic-site":
> {"gateway_recover_after_data_nodes": 1, "index_number_of_replicas": 1,
> "index_number_of_shards": 2, "network_host": "[ _local_, _eth0_ ]",
> "zen_discovery_ping_unicast_hosts": "[
> ec2-35-164-136-88.us-west-2.compute.amazonaws.com,
> ec2-34-208-94-45.us-west-2.compute.amazonaws.com ]"}}], "host":
> "ec2-35-164-26-255.us-west-2.compute.amazonaws.com", "password":
> "admin", "port": 8080, "username": "admin", "wait_for_complete":
> true}, "module_name": "ambari_cluster_state"}, "msg": "Ambari client
> exception occurred: Could not create cluster: request code 400,
> request message {\n \"status\" : 400,\n \"message\" :
> \"Invalid Cluster Creation Template:
> org.apache.ambari.server.topology.InvalidTopologyTemplateException:
> Host group 'slaves' must contain at least one 'hosts/fqdn' or a
> 'host_count' value\"\n}"}
>
> PLAY RECAP
> *********************************************************************
> ec2-34-208-94-45.us-west-2.compute.amazonaws.com : ok=49 changed=27
> unreachable=0 failed=0
> ec2-34-209-224-190.us-west-2.compute.amazonaws.com : ok=23 changed=8
> unreachable=0 failed=0
> ec2-34-210-151-29.us-west-2.compute.amazonaws.com : ok=58 changed=31
> unreachable=0 failed=0
> ec2-35-164-136-88.us-west-2.compute.amazonaws.com : ok=49 changed=27
> unreachable=0 failed=0
> ec2-35-164-26-255.us-west-2.compute.amazonaws.com : ok=54 changed=28
> unreachable=0 failed=1
> localhost : ok=29 changed=12 unreachable=0
> failed=0
>
> Any idea what might be going on?
>
> (Don't worry about the URLs, the instances have been terminated)