You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Stephen Hartzell <ha...@gmail.com> on 2014/09/09 00:01:19 UTC
How Do Workers Connect To Nimbus
All,
I would greatly appreciate any help that anyone would afford. I've been
trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
machines. So far, I haven't been able to get a cluster built. I can get a
supervisor and nimbus to run on a single machine, but I can't figure out
how to get another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm configuration file?
I've gone through many tutorials and the official documentation, but this
point doesn't seem to be covered anywhere in sufficient detail for a new
guy like me.
Some of you may be tempted to point me toward storm-deploy, but I spent
four days trying to get that to work until I gave up. I'm having Issue #58
on github. Following the instructions exactly and other tutorials on a bran
new AWS machine fails. So I gave up on storm-deploy and decided to try and
setup a cluster manually. Thanks in advance to anyone willing to offer me
any inputs you can!
Re: How Do Workers Connect To Nimbus
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Yes, it would help others. :)
On Tue, Sep 9, 2014 at 6:40 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> I'm going to make a youtube tutorial to show how I got it to work. If
> nothing else, it'll be for my reference.
>
> -Thanks, Stephen
>
> On Tue, Sep 9, 2014 at 9:07 AM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> Good to know that it solved your problem. :)
>> This is what open source community do. :)
>> Even in this mail thread, I also learned and you contributed too. You
>> shared the detailed steps to run the storm cluster. :)
>>
>>
>> On Tue, Sep 9, 2014 at 6:28 PM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> HA HA HA HA!!!! Vikas, you crazy genius!!!!!!!!!!! It works!!!! It
>>> works!!!!!!! :D :D :D :D. How much beer do I owe you? After hours and hours
>>> and hours of work... it was a port issue... Thank you so much everyone for
>>> your help and input!
>>>
>>> On Tue, Sep 9, 2014 at 8:27 AM, Vikas Agarwal <vi...@infoobjects.com>
>>> wrote:
>>>
>>>> Did you open following ports in Ec2 security group:
>>>>
>>>> 2181
>>>> 3772
>>>> 3773
>>>> 6627
>>>> 6700
>>>> 6701
>>>> 6702
>>>> 6703
>>>>
>>>> Some of the ports might not be required to be opened, but I guess try
>>>> opening these ports for the same security group not for the world and start
>>>> second instance within same security group.
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 5:46 PM, Stephen Hartzell <
>>>> hartzell.stephen@gmail.com> wrote:
>>>>
>>>>> My apologies, I run "bin/storm supervisor &" at the end. That was a
>>>>> bad copy and paste.
>>>>>
>>>>> On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com>
>>>>> wrote:
>>>>>
>>>>>> >>*I launch another basic EC2 CentOS machine and do the exact same
>>>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>>>> nimbus &" at the end*<<
>>>>>>
>>>>>> Why would you run nimbus on two machines?
>>>>>>
>>>>>> Yes, by default Hortonworks provides one supervisor node par machine,
>>>>>> so I am trying to add new EC2 machine with supervisor role. And it worked
>>>>>> upto some extent but still not fully functional. I am still testing its
>>>>>> linear scalability.
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>
>>>>>>> Vikas,
>>>>>>>
>>>>>>>
>>>>>>> I've tried to use the HortonWorks distribution, but that only
>>>>>>> provides one supervisor and nimbus on one virtual machine. I'm excited to
>>>>>>> hear that you have storm working on AWS EC2 machines because that is
>>>>>>> exactly what I am trying to do! Right now we're still in the development
>>>>>>> stage, so all we are trying to do is to have one worker machine connect to
>>>>>>> one nimbus machine. So far we haven't got this work.
>>>>>>>
>>>>>>> Although it might be lengthy, let me go ahead and post the
>>>>>>> commands I'm using to setup the nimbus machine.
>>>>>>>
>>>>>>>
>>>>>>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open
>>>>>>> to TCP connections (it's public IP is 54.68.149.181)*
>>>>>>>
>>>>>>> sudo yum update
>>>>>>>
>>>>>>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>>>>>>
>>>>>>> *# Install zeromq*
>>>>>>>
>>>>>>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>>>>>>
>>>>>>> tar –zxvf zeromq-2.1.7.tar.gz
>>>>>>>
>>>>>>> cd zeromq-2.1.7.tar.gz
>>>>>>>
>>>>>>> ./configure
>>>>>>>
>>>>>>> make
>>>>>>>
>>>>>>> sudo make install
>>>>>>>
>>>>>>> cd ~
>>>>>>>
>>>>>>> *# Install jzmq*
>>>>>>>
>>>>>>> sudo yum install git java-devel libtool
>>>>>>>
>>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>>>>>>> JDK required for jzmq in the configuration stage*
>>>>>>>
>>>>>>> cd $JAVA_HOME
>>>>>>>
>>>>>>> sudo ln –s include include *#
>>>>>>> JDK include directory has headers required for jzmq*
>>>>>>>
>>>>>>> cd ~
>>>>>>>
>>>>>>> git clone https://github.com/nathanmarz/jzmq.git
>>>>>>>
>>>>>>> cd jzmq/src/
>>>>>>>
>>>>>>> CLASSPATH=.:./.:$CLASSPATH
>>>>>>>
>>>>>>> touch classdist_noinst.stamp
>>>>>>>
>>>>>>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>>>>>>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>>>>>>> org/zeromq/ZMQStreamer.java
>>>>>>>
>>>>>>> cd ~/jzmq
>>>>>>>
>>>>>>> ./autogen.sh
>>>>>>>
>>>>>>> ./configure
>>>>>>>
>>>>>>> make
>>>>>>>
>>>>>>> sudo make install
>>>>>>>
>>>>>>> cd ~
>>>>>>>
>>>>>>> *# Download zookeeper*
>>>>>>>
>>>>>>> wget
>>>>>>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>>>>>>
>>>>>>> tar -zxvf zookeeper-3.4.6.tar.gz
>>>>>>>
>>>>>>> cd zookeeper-3.4.6
>>>>>>>
>>>>>>> vi conf/zoo.cfg
>>>>>>>
>>>>>>>
>>>>>>> tickTime=2000
>>>>>>>
>>>>>>> dataDir=/tmp/zookeeper
>>>>>>>
>>>>>>> clientPort=2181
>>>>>>>
>>>>>>>
>>>>>>> mkdir /tmp/zookeeper
>>>>>>> bin/zkServer.sh start
>>>>>>> bin/zkCli.sh -server 54.68.149.181:2181
>>>>>>>
>>>>>>>
>>>>>>> *# Download storm and modify configuration file (changes over
>>>>>>> default.yaml shown in bold)*
>>>>>>>
>>>>>>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>>>>>>> storm-0.8.2.zip
>>>>>>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>>>>>>
>>>>>>> cd storm-0.8.2.zip
>>>>>>>
>>>>>>> vi conf/storm.yaml
>>>>>>>
>>>>>>>
>>>>>>> # Licensed to the Apache Software Foundation (ASF) under one
>>>>>>>>
>>>>>>>> # or more contributor license agreements. See the NOTICE file
>>>>>>>>
>>>>>>>> # distributed with this work for additional information
>>>>>>>>
>>>>>>>> # regarding copyright ownership. The ASF licenses this file
>>>>>>>>
>>>>>>>> # to you under the Apache License, Version 2.0 (the
>>>>>>>>
>>>>>>>> # "License"); you may not use this file except in compliance
>>>>>>>>
>>>>>>>> # with the License. You may obtain a copy of the License at
>>>>>>>>
>>>>>>>> #
>>>>>>>>
>>>>>>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>>>>>>
>>>>>>>> #
>>>>>>>>
>>>>>>>> # Unless required by applicable law or agreed to in writing,
>>>>>>>>> software
>>>>>>>>
>>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>>>
>>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>>>>> implied.
>>>>>>>>
>>>>>>>> # See the License for the specific language governing permissions
>>>>>>>>> and
>>>>>>>>
>>>>>>>> # limitations under the License.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> ########### These all have default values as shown
>>>>>>>>
>>>>>>>> ########### Additional configuration goes into storm.yaml
>>>>>>>>
>>>>>>>>
>>>>>>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>>>>>>> */home/ec2-user*"
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### storm.* configs are general configurations
>>>>>>>>
>>>>>>>> # the local dir is where jars are kept
>>>>>>>>
>>>>>>>> storm.local.dir: "storm-local"
>>>>>>>>
>>>>>>>> storm.zookeeper.servers:
>>>>>>>>
>>>>>>>> - "*54.68.149.181*"
>>>>>>>>
>>>>>>>> storm.zookeeper.port: 2181
>>>>>>>>
>>>>>>>> storm.zookeeper.root: "/storm"
>>>>>>>>
>>>>>>>> storm.zookeeper.session.timeout: 20000
>>>>>>>>
>>>>>>>> storm.zookeeper.connection.timeout: 15000
>>>>>>>>
>>>>>>>> storm.zookeeper.retry.times: 5
>>>>>>>>
>>>>>>>> storm.zookeeper.retry.interval: 1000
>>>>>>>>
>>>>>>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>>>>>>
>>>>>>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>>>>>>
>>>>>>>> storm.local.mode.zmq: false
>>>>>>>>
>>>>>>>> storm.thrift.transport:
>>>>>>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>>>>>>
>>>>>>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### nimbus.* configs are for the master
>>>>>>>>
>>>>>>>> nimbus.host: "*54.68.149.181*"
>>>>>>>>
>>>>>>>> nimbus.thrift.port: 6627
>>>>>>>>
>>>>>>>> nimbus.childopts: "-Xmx1024m"
>>>>>>>>
>>>>>>>> nimbus.task.timeout.secs: 30
>>>>>>>>
>>>>>>>> nimbus.supervisor.timeout.secs: 60
>>>>>>>>
>>>>>>>> nimbus.monitor.freq.secs: 10
>>>>>>>>
>>>>>>>> nimbus.cleanup.inbox.freq.secs: 600
>>>>>>>>
>>>>>>>> nimbus.inbox.jar.expiration.secs: 3600
>>>>>>>>
>>>>>>>> nimbus.task.launch.secs: 120
>>>>>>>>
>>>>>>>> nimbus.reassign: true
>>>>>>>>
>>>>>>>> nimbus.file.copy.expiration.secs: 600
>>>>>>>>
>>>>>>>> nimbus.topology.validator:
>>>>>>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### ui.* configs are for the master
>>>>>>>>
>>>>>>>> ui.port: 8080
>>>>>>>>
>>>>>>>> ui.childopts: "-Xmx768m"
>>>>>>>>
>>>>>>>>
>>>>>>>>> logviewer.port: 8000
>>>>>>>>
>>>>>>>> logviewer.childopts: "-Xmx128m"
>>>>>>>>
>>>>>>>> logviewer.appender.name: "A1"
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> drpc.port: 3772
>>>>>>>>
>>>>>>>> drpc.worker.threads: 64
>>>>>>>>
>>>>>>>> drpc.queue.size: 128
>>>>>>>>
>>>>>>>> drpc.invocations.port: 3773
>>>>>>>>
>>>>>>>> drpc.request.timeout.secs: 600
>>>>>>>>
>>>>>>>> drpc.childopts: "-Xmx768m"
>>>>>>>>
>>>>>>>>
>>>>>>>>> transactional.zookeeper.root: "/transactional"
>>>>>>>>
>>>>>>>> transactional.zookeeper.servers: null
>>>>>>>>
>>>>>>>> transactional.zookeeper.port: null
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### supervisor.* configs are for node supervisors
>>>>>>>>
>>>>>>>> # Define the amount of workers that can be run on this machine.
>>>>>>>>> Each worker is assigned a port to use for communication
>>>>>>>>
>>>>>>>> supervisor.slots.ports:
>>>>>>>>
>>>>>>>> - 6700
>>>>>>>>
>>>>>>>> - 6701
>>>>>>>>
>>>>>>>> - 6702
>>>>>>>>
>>>>>>>> - 6703
>>>>>>>>
>>>>>>>> supervisor.childopts: "-Xmx256m"
>>>>>>>>
>>>>>>>> #how long supervisor will wait to ensure that a worker process is
>>>>>>>>> started
>>>>>>>>
>>>>>>>> supervisor.worker.start.timeout.secs: 120
>>>>>>>>
>>>>>>>> #how long between heartbeats until supervisor considers that worker
>>>>>>>>> dead and tries to restart it
>>>>>>>>
>>>>>>>> supervisor.worker.timeout.secs: 30
>>>>>>>>
>>>>>>>> #how frequently the supervisor checks on the status of the
>>>>>>>>> processes it's monitoring and restarts if necessary
>>>>>>>>
>>>>>>>> supervisor.monitor.frequency.secs: 3
>>>>>>>>
>>>>>>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>>>>>>> nimbus)
>>>>>>>>
>>>>>>>> supervisor.heartbeat.frequency.secs: 5
>>>>>>>>
>>>>>>>> supervisor.enable: true
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### worker.* configs are for task workers
>>>>>>>>
>>>>>>>> worker.childopts: "-Xmx768m"
>>>>>>>>
>>>>>>>> worker.heartbeat.frequency.secs: 1
>>>>>>>>
>>>>>>>>
>>>>>>>>> task.heartbeat.frequency.secs: 3
>>>>>>>>
>>>>>>>> task.refresh.poll.secs: 10
>>>>>>>>
>>>>>>>>
>>>>>>>>> zmq.threads: 1
>>>>>>>>
>>>>>>>> zmq.linger.millis: 5000
>>>>>>>>
>>>>>>>> zmq.hwm: 0
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> storm.messaging.netty.server_worker_threads: 1
>>>>>>>>
>>>>>>>> storm.messaging.netty.client_worker_threads: 1
>>>>>>>>
>>>>>>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>>>>>>
>>>>>>>> storm.messaging.netty.max_retries: 30
>>>>>>>>
>>>>>>>> storm.messaging.netty.max_wait_ms: 1000
>>>>>>>>
>>>>>>>> storm.messaging.netty.min_wait_ms: 100
>>>>>>>>
>>>>>>>>
>>>>>>>>> ### topology.* configs are for specific executing storms
>>>>>>>>
>>>>>>>> topology.enable.message.timeouts: true
>>>>>>>>
>>>>>>>> topology.debug: false
>>>>>>>>
>>>>>>>> topology.optimize: true
>>>>>>>>
>>>>>>>> topology.workers: 1
>>>>>>>>
>>>>>>>> topology.acker.executors: null
>>>>>>>>
>>>>>>>> topology.tasks: null
>>>>>>>>
>>>>>>>> # maximum amount of time a message has to complete before it's
>>>>>>>>> considered failed
>>>>>>>>
>>>>>>>> topology.message.timeout.secs: 30
>>>>>>>>
>>>>>>>> topology.skip.missing.kryo.registrations: false
>>>>>>>>
>>>>>>>> topology.max.task.parallelism: null
>>>>>>>>
>>>>>>>> topology.max.spout.pending: null
>>>>>>>>
>>>>>>>> topology.state.synchronization.timeout.secs: 60
>>>>>>>>
>>>>>>>> topology.stats.sample.rate: 0.05
>>>>>>>>
>>>>>>>> topology.builtin.metrics.bucket.size.secs: 60
>>>>>>>>
>>>>>>>> topology.fall.back.on.java.serialization: true
>>>>>>>>
>>>>>>>> topology.worker.childopts: null
>>>>>>>>
>>>>>>>> topology.executor.receive.buffer.size: 1024 #batched
>>>>>>>>
>>>>>>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>>>>>>
>>>>>>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot
>>>>>>>>> of problems (heartbeat thread gets starved, throughput plummets)
>>>>>>>>
>>>>>>>> topology.transfer.buffer.size: 1024 # batched
>>>>>>>>
>>>>>>>> topology.tick.tuple.freq.secs: null
>>>>>>>>
>>>>>>>> topology.worker.shared.thread.pool.size: 4
>>>>>>>>
>>>>>>>> topology.disruptor.wait.strategy:
>>>>>>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>>>>>>
>>>>>>>> topology.spout.wait.strategy:
>>>>>>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>>>>>>
>>>>>>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>>>>>>
>>>>>>>> topology.error.throttle.interval.secs: 10
>>>>>>>>
>>>>>>>> topology.max.error.report.per.interval: 5
>>>>>>>>
>>>>>>>> topology.kryo.factory:
>>>>>>>>> "backtype.storm.serialization.DefaultKryoFactory"
>>>>>>>>
>>>>>>>> topology.tuple.serializer:
>>>>>>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>>>>>>
>>>>>>>> topology.trident.batch.emit.interval.millis: 500
>>>>>>>>
>>>>>>>>
>>>>>>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> bin/storm nimbus &
>>>>>>>
>>>>>>> bin/storm supervisor &
>>>>>>>
>>>>>>> bin/storm ui &
>>>>>>>
>>>>>>> *I launch another basic EC2 CentOS machine and do the exact same
>>>>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>>>>> nimbus &" at the end*
>>>>>>>
>>>>>>> Any thoughts would be greatly appreciated. Thanks so much for all of
>>>>>>> your help. I'm sure someone else has done this before!
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <
>>>>>>> vikas@infoobjects.com> wrote:
>>>>>>>
>>>>>>>> Although, implementing the Storm cluster manually would be really
>>>>>>>> nice learning, I would suggest using HortonWorks distribution which comes
>>>>>>>> with Storm as OOTB solution and you can configure everything from Ambari
>>>>>>>> UI. We are using Storm on Amazon EC2 machine, though it is right now in
>>>>>>>> beta stage. We are going to move to production in coming 2-3 months.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> All,
>>>>>>>>>
>>>>>>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>>>>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>>>>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>>>>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>>>>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>>>
>>>>>>>>> You should be able to see the error it gives by going to the web
>>>>>>>>> page yourself at: http://54.68.149.181:8080. I am only using this
>>>>>>>>> account to test and see if I can even get storm to work, so these machines
>>>>>>>>> are only for testing. Perhaps someone could tell me what the storm.yaml
>>>>>>>>> file should look like for this setup?
>>>>>>>>>
>>>>>>>>> -Thanks, Stephne
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I'm getting kind of confused by the storm.yaml file. Should I be
>>>>>>>>>> using the default.yaml and just modify the zookeeper and nimbus ip, or
>>>>>>>>>> should I use a bran new storm.yaml?
>>>>>>>>>>
>>>>>>>>>> My nimbus machine has the ip address: 54.68.149.181. My
>>>>>>>>>> zookeeper is on the nimbus machine. what should the storm.yaml look like on
>>>>>>>>>> my worker and nimbus machine? Will the storm.yaml be the same on my worker
>>>>>>>>>> and nimbus machine? I am not trying to do anything fancy, I am just trying
>>>>>>>>>> to get a very basic cluster up and running.
>>>>>>>>>>
>>>>>>>>>> -Thanks, Stephen
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>>>>>>> nothing got it working for me!
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> The worker connects to the thrift port and not the ui port. You
>>>>>>>>>>>> need to open port 6627 or whatever is the value being set in storm.yaml
>>>>>>>>>>>> using property “nimbus.thrift.port”.
>>>>>>>>>>>>
>>>>>>>>>>>> Based on the configuration that you have pointed so far it
>>>>>>>>>>>> seems your nimbus host has nimbus,ui,supervisor working because you
>>>>>>>>>>>> actually have zookeeper running locally on that host. As Harsha pointed out
>>>>>>>>>>>> you need to change it to a value that is the public ip instead of loopback
>>>>>>>>>>>> interface.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Parth
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a
>>>>>>>>>>>> loopback interface , instead either use your public ip or 0.0.0.0*
>>>>>>>>>>>> )
>>>>>>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>>>>>>> machine is restarted)*
>>>>>>>>>>>>
>>>>>>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public
>>>>>>>>>>>> ip not 127.0.0.1.
>>>>>>>>>>>>
>>>>>>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>>>>>>> 54.68.149.181"
>>>>>>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>>>>>>> access the UI using it?
>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Harsha,
>>>>>>>>>>>>
>>>>>>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>>>>>>
>>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>>>>>>
>>>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>>>>>>
>>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>>> - "54.68.149.181"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>>>>>>
>>>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>>>
>>>>>>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>>>>>>> host.
>>>>>>>>>>>> -Thanks, Stephen
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> There aren't any errors in worker machine supervisor logs. Are
>>>>>>>>>>>> you using the same storm.yaml for both the machines and also are you able
>>>>>>>>>>>> to ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Harsha,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks so much for getting back with me. I will check the
>>>>>>>>>>>> logs, but I don't seem to get any error messages. I have a nimbus AWS
>>>>>>>>>>>> machine with zookeeper on it and a worker AWS machine.
>>>>>>>>>>>>
>>>>>>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>>>>>>
>>>>>>>>>>>> bin/storm nimbus &
>>>>>>>>>>>> bin/storm supervisor &
>>>>>>>>>>>> bin/storm ui
>>>>>>>>>>>>
>>>>>>>>>>>> On the worker machine I run:
>>>>>>>>>>>> bin/storm supervisor
>>>>>>>>>>>>
>>>>>>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on
>>>>>>>>>>>> the nimbus machine). So apparently, the worker machine isn't "registering"
>>>>>>>>>>>> with the nimbus machine.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Stephen,
>>>>>>>>>>>> What are the issues you are seeing.
>>>>>>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it
>>>>>>>>>>>> in the storm configuration file"
>>>>>>>>>>>>
>>>>>>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are
>>>>>>>>>>>> able to connect to your zookeeper cluster.
>>>>>>>>>>>> Check your logs under storm_inst/logs/ for any errors when you
>>>>>>>>>>>> try to start nimbus or supervisors.
>>>>>>>>>>>> If you are installing it manually try following these steps if
>>>>>>>>>>>> you are not already done.
>>>>>>>>>>>>
>>>>>>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>>>>>>> -Harsha
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> All,
>>>>>>>>>>>>
>>>>>>>>>>>> I would greatly appreciate any help that anyone would afford.
>>>>>>>>>>>> I've been trying to setup a storm cluster on AWS for a few weeks now on
>>>>>>>>>>>> centOS EC2 machines. So far, I haven't been able to get a cluster built. I
>>>>>>>>>>>> can get a supervisor and nimbus to run on a single machine, but I can't
>>>>>>>>>>>> figure out how to get another worker to connect to nimbus. How do worker
>>>>>>>>>>>> machines "know" how to connect to nimbus? Is it in the storm configuration
>>>>>>>>>>>> file? I've gone through many tutorials and the official documentation, but
>>>>>>>>>>>> this point doesn't seem to be covered anywhere in sufficient detail for a
>>>>>>>>>>>> new guy like me.
>>>>>>>>>>>>
>>>>>>>>>>>> Some of you may be tempted to point me toward storm-deploy,
>>>>>>>>>>>> but I spent four days trying to get that to work until I gave up. I'm
>>>>>>>>>>>> having Issue #58 on github. Following the instructions exactly and other
>>>>>>>>>>>> tutorials on a bran new AWS machine fails. So I gave up on storm-deploy and
>>>>>>>>>>>> decided to try and setup a cluster manually. Thanks in advance to anyone
>>>>>>>>>>>> willing to offer me any inputs you can!
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>>>>>>>>>>> or entity to which it is addressed and may contain information that is
>>>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Vikas Agarwal
>>>>>>>> 91 – 9928301411
>>>>>>>>
>>>>>>>> InfoObjects, Inc.
>>>>>>>> Execution Matters
>>>>>>>> http://www.infoobjects.com
>>>>>>>> 2041 Mission College Boulevard, #280
>>>>>>>> Santa Clara, CA 95054
>>>>>>>> +1 (408) 988-2000 Work
>>>>>>>> +1 (408) 716-2726 Fax
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Vikas Agarwal
>>>>>> 91 – 9928301411
>>>>>>
>>>>>> InfoObjects, Inc.
>>>>>> Execution Matters
>>>>>> http://www.infoobjects.com
>>>>>> 2041 Mission College Boulevard, #280
>>>>>> Santa Clara, CA 95054
>>>>>> +1 (408) 988-2000 Work
>>>>>> +1 (408) 716-2726 Fax
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Vikas Agarwal
>>>> 91 – 9928301411
>>>>
>>>> InfoObjects, Inc.
>>>> Execution Matters
>>>> http://www.infoobjects.com
>>>> 2041 Mission College Boulevard, #280
>>>> Santa Clara, CA 95054
>>>> +1 (408) 988-2000 Work
>>>> +1 (408) 716-2726 Fax
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
I'm going to make a youtube tutorial to show how I got it to work. If
nothing else, it'll be for my reference.
-Thanks, Stephen
On Tue, Sep 9, 2014 at 9:07 AM, Vikas Agarwal <vi...@infoobjects.com> wrote:
> Good to know that it solved your problem. :)
> This is what open source community do. :)
> Even in this mail thread, I also learned and you contributed too. You
> shared the detailed steps to run the storm cluster. :)
>
>
> On Tue, Sep 9, 2014 at 6:28 PM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> HA HA HA HA!!!! Vikas, you crazy genius!!!!!!!!!!! It works!!!! It
>> works!!!!!!! :D :D :D :D. How much beer do I owe you? After hours and hours
>> and hours of work... it was a port issue... Thank you so much everyone for
>> your help and input!
>>
>> On Tue, Sep 9, 2014 at 8:27 AM, Vikas Agarwal <vi...@infoobjects.com>
>> wrote:
>>
>>> Did you open following ports in Ec2 security group:
>>>
>>> 2181
>>> 3772
>>> 3773
>>> 6627
>>> 6700
>>> 6701
>>> 6702
>>> 6703
>>>
>>> Some of the ports might not be required to be opened, but I guess try
>>> opening these ports for the same security group not for the world and start
>>> second instance within same security group.
>>>
>>>
>>> On Tue, Sep 9, 2014 at 5:46 PM, Stephen Hartzell <
>>> hartzell.stephen@gmail.com> wrote:
>>>
>>>> My apologies, I run "bin/storm supervisor &" at the end. That was a bad
>>>> copy and paste.
>>>>
>>>> On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com>
>>>> wrote:
>>>>
>>>>> >>*I launch another basic EC2 CentOS machine and do the exact same
>>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>>> nimbus &" at the end*<<
>>>>>
>>>>> Why would you run nimbus on two machines?
>>>>>
>>>>> Yes, by default Hortonworks provides one supervisor node par machine,
>>>>> so I am trying to add new EC2 machine with supervisor role. And it worked
>>>>> upto some extent but still not fully functional. I am still testing its
>>>>> linear scalability.
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>
>>>>>> Vikas,
>>>>>>
>>>>>>
>>>>>> I've tried to use the HortonWorks distribution, but that only
>>>>>> provides one supervisor and nimbus on one virtual machine. I'm excited to
>>>>>> hear that you have storm working on AWS EC2 machines because that is
>>>>>> exactly what I am trying to do! Right now we're still in the development
>>>>>> stage, so all we are trying to do is to have one worker machine connect to
>>>>>> one nimbus machine. So far we haven't got this work.
>>>>>>
>>>>>> Although it might be lengthy, let me go ahead and post the commands
>>>>>> I'm using to setup the nimbus machine.
>>>>>>
>>>>>>
>>>>>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to
>>>>>> TCP connections (it's public IP is 54.68.149.181)*
>>>>>>
>>>>>> sudo yum update
>>>>>>
>>>>>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>>>>>
>>>>>> *# Install zeromq*
>>>>>>
>>>>>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>>>>>
>>>>>> tar –zxvf zeromq-2.1.7.tar.gz
>>>>>>
>>>>>> cd zeromq-2.1.7.tar.gz
>>>>>>
>>>>>> ./configure
>>>>>>
>>>>>> make
>>>>>>
>>>>>> sudo make install
>>>>>>
>>>>>> cd ~
>>>>>>
>>>>>> *# Install jzmq*
>>>>>>
>>>>>> sudo yum install git java-devel libtool
>>>>>>
>>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>>>>>> JDK required for jzmq in the configuration stage*
>>>>>>
>>>>>> cd $JAVA_HOME
>>>>>>
>>>>>> sudo ln –s include include *#
>>>>>> JDK include directory has headers required for jzmq*
>>>>>>
>>>>>> cd ~
>>>>>>
>>>>>> git clone https://github.com/nathanmarz/jzmq.git
>>>>>>
>>>>>> cd jzmq/src/
>>>>>>
>>>>>> CLASSPATH=.:./.:$CLASSPATH
>>>>>>
>>>>>> touch classdist_noinst.stamp
>>>>>>
>>>>>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>>>>>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>>>>>> org/zeromq/ZMQStreamer.java
>>>>>>
>>>>>> cd ~/jzmq
>>>>>>
>>>>>> ./autogen.sh
>>>>>>
>>>>>> ./configure
>>>>>>
>>>>>> make
>>>>>>
>>>>>> sudo make install
>>>>>>
>>>>>> cd ~
>>>>>>
>>>>>> *# Download zookeeper*
>>>>>>
>>>>>> wget
>>>>>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>>>>>
>>>>>> tar -zxvf zookeeper-3.4.6.tar.gz
>>>>>>
>>>>>> cd zookeeper-3.4.6
>>>>>>
>>>>>> vi conf/zoo.cfg
>>>>>>
>>>>>>
>>>>>> tickTime=2000
>>>>>>
>>>>>> dataDir=/tmp/zookeeper
>>>>>>
>>>>>> clientPort=2181
>>>>>>
>>>>>>
>>>>>> mkdir /tmp/zookeeper
>>>>>> bin/zkServer.sh start
>>>>>> bin/zkCli.sh -server 54.68.149.181:2181
>>>>>>
>>>>>>
>>>>>> *# Download storm and modify configuration file (changes over
>>>>>> default.yaml shown in bold)*
>>>>>>
>>>>>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>>>>>> storm-0.8.2.zip
>>>>>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>>>>>
>>>>>> cd storm-0.8.2.zip
>>>>>>
>>>>>> vi conf/storm.yaml
>>>>>>
>>>>>>
>>>>>> # Licensed to the Apache Software Foundation (ASF) under one
>>>>>>>
>>>>>>> # or more contributor license agreements. See the NOTICE file
>>>>>>>
>>>>>>> # distributed with this work for additional information
>>>>>>>
>>>>>>> # regarding copyright ownership. The ASF licenses this file
>>>>>>>
>>>>>>> # to you under the Apache License, Version 2.0 (the
>>>>>>>
>>>>>>> # "License"); you may not use this file except in compliance
>>>>>>>
>>>>>>> # with the License. You may obtain a copy of the License at
>>>>>>>
>>>>>>> #
>>>>>>>
>>>>>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>>>>>
>>>>>>> #
>>>>>>>
>>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>>
>>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>>
>>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>>>> implied.
>>>>>>>
>>>>>>> # See the License for the specific language governing permissions and
>>>>>>>
>>>>>>> # limitations under the License.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> ########### These all have default values as shown
>>>>>>>
>>>>>>> ########### Additional configuration goes into storm.yaml
>>>>>>>
>>>>>>>
>>>>>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>>>>>> */home/ec2-user*"
>>>>>>>
>>>>>>>
>>>>>>>> ### storm.* configs are general configurations
>>>>>>>
>>>>>>> # the local dir is where jars are kept
>>>>>>>
>>>>>>> storm.local.dir: "storm-local"
>>>>>>>
>>>>>>> storm.zookeeper.servers:
>>>>>>>
>>>>>>> - "*54.68.149.181*"
>>>>>>>
>>>>>>> storm.zookeeper.port: 2181
>>>>>>>
>>>>>>> storm.zookeeper.root: "/storm"
>>>>>>>
>>>>>>> storm.zookeeper.session.timeout: 20000
>>>>>>>
>>>>>>> storm.zookeeper.connection.timeout: 15000
>>>>>>>
>>>>>>> storm.zookeeper.retry.times: 5
>>>>>>>
>>>>>>> storm.zookeeper.retry.interval: 1000
>>>>>>>
>>>>>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>>>>>
>>>>>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>>>>>
>>>>>>> storm.local.mode.zmq: false
>>>>>>>
>>>>>>> storm.thrift.transport:
>>>>>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>>>>>
>>>>>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>>>>>
>>>>>>>
>>>>>>>> ### nimbus.* configs are for the master
>>>>>>>
>>>>>>> nimbus.host: "*54.68.149.181*"
>>>>>>>
>>>>>>> nimbus.thrift.port: 6627
>>>>>>>
>>>>>>> nimbus.childopts: "-Xmx1024m"
>>>>>>>
>>>>>>> nimbus.task.timeout.secs: 30
>>>>>>>
>>>>>>> nimbus.supervisor.timeout.secs: 60
>>>>>>>
>>>>>>> nimbus.monitor.freq.secs: 10
>>>>>>>
>>>>>>> nimbus.cleanup.inbox.freq.secs: 600
>>>>>>>
>>>>>>> nimbus.inbox.jar.expiration.secs: 3600
>>>>>>>
>>>>>>> nimbus.task.launch.secs: 120
>>>>>>>
>>>>>>> nimbus.reassign: true
>>>>>>>
>>>>>>> nimbus.file.copy.expiration.secs: 600
>>>>>>>
>>>>>>> nimbus.topology.validator:
>>>>>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>>>>>
>>>>>>>
>>>>>>>> ### ui.* configs are for the master
>>>>>>>
>>>>>>> ui.port: 8080
>>>>>>>
>>>>>>> ui.childopts: "-Xmx768m"
>>>>>>>
>>>>>>>
>>>>>>>> logviewer.port: 8000
>>>>>>>
>>>>>>> logviewer.childopts: "-Xmx128m"
>>>>>>>
>>>>>>> logviewer.appender.name: "A1"
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> drpc.port: 3772
>>>>>>>
>>>>>>> drpc.worker.threads: 64
>>>>>>>
>>>>>>> drpc.queue.size: 128
>>>>>>>
>>>>>>> drpc.invocations.port: 3773
>>>>>>>
>>>>>>> drpc.request.timeout.secs: 600
>>>>>>>
>>>>>>> drpc.childopts: "-Xmx768m"
>>>>>>>
>>>>>>>
>>>>>>>> transactional.zookeeper.root: "/transactional"
>>>>>>>
>>>>>>> transactional.zookeeper.servers: null
>>>>>>>
>>>>>>> transactional.zookeeper.port: null
>>>>>>>
>>>>>>>
>>>>>>>> ### supervisor.* configs are for node supervisors
>>>>>>>
>>>>>>> # Define the amount of workers that can be run on this machine. Each
>>>>>>>> worker is assigned a port to use for communication
>>>>>>>
>>>>>>> supervisor.slots.ports:
>>>>>>>
>>>>>>> - 6700
>>>>>>>
>>>>>>> - 6701
>>>>>>>
>>>>>>> - 6702
>>>>>>>
>>>>>>> - 6703
>>>>>>>
>>>>>>> supervisor.childopts: "-Xmx256m"
>>>>>>>
>>>>>>> #how long supervisor will wait to ensure that a worker process is
>>>>>>>> started
>>>>>>>
>>>>>>> supervisor.worker.start.timeout.secs: 120
>>>>>>>
>>>>>>> #how long between heartbeats until supervisor considers that worker
>>>>>>>> dead and tries to restart it
>>>>>>>
>>>>>>> supervisor.worker.timeout.secs: 30
>>>>>>>
>>>>>>> #how frequently the supervisor checks on the status of the processes
>>>>>>>> it's monitoring and restarts if necessary
>>>>>>>
>>>>>>> supervisor.monitor.frequency.secs: 3
>>>>>>>
>>>>>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>>>>>> nimbus)
>>>>>>>
>>>>>>> supervisor.heartbeat.frequency.secs: 5
>>>>>>>
>>>>>>> supervisor.enable: true
>>>>>>>
>>>>>>>
>>>>>>>> ### worker.* configs are for task workers
>>>>>>>
>>>>>>> worker.childopts: "-Xmx768m"
>>>>>>>
>>>>>>> worker.heartbeat.frequency.secs: 1
>>>>>>>
>>>>>>>
>>>>>>>> task.heartbeat.frequency.secs: 3
>>>>>>>
>>>>>>> task.refresh.poll.secs: 10
>>>>>>>
>>>>>>>
>>>>>>>> zmq.threads: 1
>>>>>>>
>>>>>>> zmq.linger.millis: 5000
>>>>>>>
>>>>>>> zmq.hwm: 0
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> storm.messaging.netty.server_worker_threads: 1
>>>>>>>
>>>>>>> storm.messaging.netty.client_worker_threads: 1
>>>>>>>
>>>>>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>>>>>
>>>>>>> storm.messaging.netty.max_retries: 30
>>>>>>>
>>>>>>> storm.messaging.netty.max_wait_ms: 1000
>>>>>>>
>>>>>>> storm.messaging.netty.min_wait_ms: 100
>>>>>>>
>>>>>>>
>>>>>>>> ### topology.* configs are for specific executing storms
>>>>>>>
>>>>>>> topology.enable.message.timeouts: true
>>>>>>>
>>>>>>> topology.debug: false
>>>>>>>
>>>>>>> topology.optimize: true
>>>>>>>
>>>>>>> topology.workers: 1
>>>>>>>
>>>>>>> topology.acker.executors: null
>>>>>>>
>>>>>>> topology.tasks: null
>>>>>>>
>>>>>>> # maximum amount of time a message has to complete before it's
>>>>>>>> considered failed
>>>>>>>
>>>>>>> topology.message.timeout.secs: 30
>>>>>>>
>>>>>>> topology.skip.missing.kryo.registrations: false
>>>>>>>
>>>>>>> topology.max.task.parallelism: null
>>>>>>>
>>>>>>> topology.max.spout.pending: null
>>>>>>>
>>>>>>> topology.state.synchronization.timeout.secs: 60
>>>>>>>
>>>>>>> topology.stats.sample.rate: 0.05
>>>>>>>
>>>>>>> topology.builtin.metrics.bucket.size.secs: 60
>>>>>>>
>>>>>>> topology.fall.back.on.java.serialization: true
>>>>>>>
>>>>>>> topology.worker.childopts: null
>>>>>>>
>>>>>>> topology.executor.receive.buffer.size: 1024 #batched
>>>>>>>
>>>>>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>>>>>
>>>>>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot
>>>>>>>> of problems (heartbeat thread gets starved, throughput plummets)
>>>>>>>
>>>>>>> topology.transfer.buffer.size: 1024 # batched
>>>>>>>
>>>>>>> topology.tick.tuple.freq.secs: null
>>>>>>>
>>>>>>> topology.worker.shared.thread.pool.size: 4
>>>>>>>
>>>>>>> topology.disruptor.wait.strategy:
>>>>>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>>>>>
>>>>>>> topology.spout.wait.strategy:
>>>>>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>>>>>
>>>>>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>>>>>
>>>>>>> topology.error.throttle.interval.secs: 10
>>>>>>>
>>>>>>> topology.max.error.report.per.interval: 5
>>>>>>>
>>>>>>> topology.kryo.factory:
>>>>>>>> "backtype.storm.serialization.DefaultKryoFactory"
>>>>>>>
>>>>>>> topology.tuple.serializer:
>>>>>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>>>>>
>>>>>>> topology.trident.batch.emit.interval.millis: 500
>>>>>>>
>>>>>>>
>>>>>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> bin/storm nimbus &
>>>>>>
>>>>>> bin/storm supervisor &
>>>>>>
>>>>>> bin/storm ui &
>>>>>>
>>>>>> *I launch another basic EC2 CentOS machine and do the exact same
>>>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>>>> nimbus &" at the end*
>>>>>>
>>>>>> Any thoughts would be greatly appreciated. Thanks so much for all of
>>>>>> your help. I'm sure someone else has done this before!
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vikas@infoobjects.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Although, implementing the Storm cluster manually would be really
>>>>>>> nice learning, I would suggest using HortonWorks distribution which comes
>>>>>>> with Storm as OOTB solution and you can configure everything from Ambari
>>>>>>> UI. We are using Storm on Amazon EC2 machine, though it is right now in
>>>>>>> beta stage. We are going to move to production in coming 2-3 months.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>>>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>>>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>>>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>>>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>>
>>>>>>>> You should be able to see the error it gives by going to the web
>>>>>>>> page yourself at: http://54.68.149.181:8080. I am only using this
>>>>>>>> account to test and see if I can even get storm to work, so these machines
>>>>>>>> are only for testing. Perhaps someone could tell me what the storm.yaml
>>>>>>>> file should look like for this setup?
>>>>>>>>
>>>>>>>> -Thanks, Stephne
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I'm getting kind of confused by the storm.yaml file. Should I be
>>>>>>>>> using the default.yaml and just modify the zookeeper and nimbus ip, or
>>>>>>>>> should I use a bran new storm.yaml?
>>>>>>>>>
>>>>>>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper
>>>>>>>>> is on the nimbus machine. what should the storm.yaml look like on my worker
>>>>>>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>>>>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>>>>>>> very basic cluster up and running.
>>>>>>>>>
>>>>>>>>> -Thanks, Stephen
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>>>>>> nothing got it working for me!
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> The worker connects to the thrift port and not the ui port. You
>>>>>>>>>>> need to open port 6627 or whatever is the value being set in storm.yaml
>>>>>>>>>>> using property “nimbus.thrift.port”.
>>>>>>>>>>>
>>>>>>>>>>> Based on the configuration that you have pointed so far it seems
>>>>>>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Parth
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>
>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>>>>>> machine is restarted)*
>>>>>>>>>>>
>>>>>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public
>>>>>>>>>>> ip not 127.0.0.1.
>>>>>>>>>>>
>>>>>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>>>>>> 54.68.149.181"
>>>>>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>>>>>> access the UI using it?
>>>>>>>>>>> -Harsha
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>>>>>
>>>>>>>>>>> Harsha,
>>>>>>>>>>>
>>>>>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>>>>>
>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>>>>>
>>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>>>>>
>>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>>> - "54.68.149.181"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>>>>>
>>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>>
>>>>>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>>>>>> host.
>>>>>>>>>>> -Thanks, Stephen
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There aren't any errors in worker machine supervisor logs. Are
>>>>>>>>>>> you using the same storm.yaml for both the machines and also are you able
>>>>>>>>>>> to ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>>>>>> -Harsha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>>>>>
>>>>>>>>>>> Harsha,
>>>>>>>>>>>
>>>>>>>>>>> Thanks so much for getting back with me. I will check the
>>>>>>>>>>> logs, but I don't seem to get any error messages. I have a nimbus AWS
>>>>>>>>>>> machine with zookeeper on it and a worker AWS machine.
>>>>>>>>>>>
>>>>>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>>>>>
>>>>>>>>>>> bin/storm nimbus &
>>>>>>>>>>> bin/storm supervisor &
>>>>>>>>>>> bin/storm ui
>>>>>>>>>>>
>>>>>>>>>>> On the worker machine I run:
>>>>>>>>>>> bin/storm supervisor
>>>>>>>>>>>
>>>>>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on
>>>>>>>>>>> the nimbus machine). So apparently, the worker machine isn't "registering"
>>>>>>>>>>> with the nimbus machine.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Stephen,
>>>>>>>>>>> What are the issues you are seeing.
>>>>>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it
>>>>>>>>>>> in the storm configuration file"
>>>>>>>>>>>
>>>>>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are
>>>>>>>>>>> able to connect to your zookeeper cluster.
>>>>>>>>>>> Check your logs under storm_inst/logs/ for any errors when you
>>>>>>>>>>> try to start nimbus or supervisors.
>>>>>>>>>>> If you are installing it manually try following these steps if
>>>>>>>>>>> you are not already done.
>>>>>>>>>>>
>>>>>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>>>>>> -Harsha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>>>>>
>>>>>>>>>>> All,
>>>>>>>>>>>
>>>>>>>>>>> I would greatly appreciate any help that anyone would afford.
>>>>>>>>>>> I've been trying to setup a storm cluster on AWS for a few weeks now on
>>>>>>>>>>> centOS EC2 machines. So far, I haven't been able to get a cluster built. I
>>>>>>>>>>> can get a supervisor and nimbus to run on a single machine, but I can't
>>>>>>>>>>> figure out how to get another worker to connect to nimbus. How do worker
>>>>>>>>>>> machines "know" how to connect to nimbus? Is it in the storm configuration
>>>>>>>>>>> file? I've gone through many tutorials and the official documentation, but
>>>>>>>>>>> this point doesn't seem to be covered anywhere in sufficient detail for a
>>>>>>>>>>> new guy like me.
>>>>>>>>>>>
>>>>>>>>>>> Some of you may be tempted to point me toward storm-deploy,
>>>>>>>>>>> but I spent four days trying to get that to work until I gave up. I'm
>>>>>>>>>>> having Issue #58 on github. Following the instructions exactly and other
>>>>>>>>>>> tutorials on a bran new AWS machine fails. So I gave up on storm-deploy and
>>>>>>>>>>> decided to try and setup a cluster manually. Thanks in advance to anyone
>>>>>>>>>>> willing to offer me any inputs you can!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>>>>>>>>>> or entity to which it is addressed and may contain information that is
>>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Vikas Agarwal
>>>>>>> 91 – 9928301411
>>>>>>>
>>>>>>> InfoObjects, Inc.
>>>>>>> Execution Matters
>>>>>>> http://www.infoobjects.com
>>>>>>> 2041 Mission College Boulevard, #280
>>>>>>> Santa Clara, CA 95054
>>>>>>> +1 (408) 988-2000 Work
>>>>>>> +1 (408) 716-2726 Fax
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Vikas Agarwal
>>>>> 91 – 9928301411
>>>>>
>>>>> InfoObjects, Inc.
>>>>> Execution Matters
>>>>> http://www.infoobjects.com
>>>>> 2041 Mission College Boulevard, #280
>>>>> Santa Clara, CA 95054
>>>>> +1 (408) 988-2000 Work
>>>>> +1 (408) 716-2726 Fax
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Vikas Agarwal
>>> 91 – 9928301411
>>>
>>> InfoObjects, Inc.
>>> Execution Matters
>>> http://www.infoobjects.com
>>> 2041 Mission College Boulevard, #280
>>> Santa Clara, CA 95054
>>> +1 (408) 988-2000 Work
>>> +1 (408) 716-2726 Fax
>>>
>>>
>>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: How Do Workers Connect To Nimbus
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Good to know that it solved your problem. :)
This is what open source community do. :)
Even in this mail thread, I also learned and you contributed too. You
shared the detailed steps to run the storm cluster. :)
On Tue, Sep 9, 2014 at 6:28 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> HA HA HA HA!!!! Vikas, you crazy genius!!!!!!!!!!! It works!!!! It
> works!!!!!!! :D :D :D :D. How much beer do I owe you? After hours and hours
> and hours of work... it was a port issue... Thank you so much everyone for
> your help and input!
>
> On Tue, Sep 9, 2014 at 8:27 AM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> Did you open following ports in Ec2 security group:
>>
>> 2181
>> 3772
>> 3773
>> 6627
>> 6700
>> 6701
>> 6702
>> 6703
>>
>> Some of the ports might not be required to be opened, but I guess try
>> opening these ports for the same security group not for the world and start
>> second instance within same security group.
>>
>>
>> On Tue, Sep 9, 2014 at 5:46 PM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> My apologies, I run "bin/storm supervisor &" at the end. That was a bad
>>> copy and paste.
>>>
>>> On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com>
>>> wrote:
>>>
>>>> >>*I launch another basic EC2 CentOS machine and do the exact same
>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>> nimbus &" at the end*<<
>>>>
>>>> Why would you run nimbus on two machines?
>>>>
>>>> Yes, by default Hortonworks provides one supervisor node par machine,
>>>> so I am trying to add new EC2 machine with supervisor role. And it worked
>>>> upto some extent but still not fully functional. I am still testing its
>>>> linear scalability.
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
>>>> hartzell.stephen@gmail.com> wrote:
>>>>
>>>>> Vikas,
>>>>>
>>>>>
>>>>> I've tried to use the HortonWorks distribution, but that only
>>>>> provides one supervisor and nimbus on one virtual machine. I'm excited to
>>>>> hear that you have storm working on AWS EC2 machines because that is
>>>>> exactly what I am trying to do! Right now we're still in the development
>>>>> stage, so all we are trying to do is to have one worker machine connect to
>>>>> one nimbus machine. So far we haven't got this work.
>>>>>
>>>>> Although it might be lengthy, let me go ahead and post the commands
>>>>> I'm using to setup the nimbus machine.
>>>>>
>>>>>
>>>>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to
>>>>> TCP connections (it's public IP is 54.68.149.181)*
>>>>>
>>>>> sudo yum update
>>>>>
>>>>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>>>>
>>>>> *# Install zeromq*
>>>>>
>>>>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>>>>
>>>>> tar –zxvf zeromq-2.1.7.tar.gz
>>>>>
>>>>> cd zeromq-2.1.7.tar.gz
>>>>>
>>>>> ./configure
>>>>>
>>>>> make
>>>>>
>>>>> sudo make install
>>>>>
>>>>> cd ~
>>>>>
>>>>> *# Install jzmq*
>>>>>
>>>>> sudo yum install git java-devel libtool
>>>>>
>>>>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>>>>> JDK required for jzmq in the configuration stage*
>>>>>
>>>>> cd $JAVA_HOME
>>>>>
>>>>> sudo ln –s include include *#
>>>>> JDK include directory has headers required for jzmq*
>>>>>
>>>>> cd ~
>>>>>
>>>>> git clone https://github.com/nathanmarz/jzmq.git
>>>>>
>>>>> cd jzmq/src/
>>>>>
>>>>> CLASSPATH=.:./.:$CLASSPATH
>>>>>
>>>>> touch classdist_noinst.stamp
>>>>>
>>>>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>>>>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>>>>> org/zeromq/ZMQStreamer.java
>>>>>
>>>>> cd ~/jzmq
>>>>>
>>>>> ./autogen.sh
>>>>>
>>>>> ./configure
>>>>>
>>>>> make
>>>>>
>>>>> sudo make install
>>>>>
>>>>> cd ~
>>>>>
>>>>> *# Download zookeeper*
>>>>>
>>>>> wget
>>>>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>>>>
>>>>> tar -zxvf zookeeper-3.4.6.tar.gz
>>>>>
>>>>> cd zookeeper-3.4.6
>>>>>
>>>>> vi conf/zoo.cfg
>>>>>
>>>>>
>>>>> tickTime=2000
>>>>>
>>>>> dataDir=/tmp/zookeeper
>>>>>
>>>>> clientPort=2181
>>>>>
>>>>>
>>>>> mkdir /tmp/zookeeper
>>>>> bin/zkServer.sh start
>>>>> bin/zkCli.sh -server 54.68.149.181:2181
>>>>>
>>>>>
>>>>> *# Download storm and modify configuration file (changes over
>>>>> default.yaml shown in bold)*
>>>>>
>>>>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>>>>> storm-0.8.2.zip
>>>>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>>>>
>>>>> cd storm-0.8.2.zip
>>>>>
>>>>> vi conf/storm.yaml
>>>>>
>>>>>
>>>>> # Licensed to the Apache Software Foundation (ASF) under one
>>>>>>
>>>>>> # or more contributor license agreements. See the NOTICE file
>>>>>>
>>>>>> # distributed with this work for additional information
>>>>>>
>>>>>> # regarding copyright ownership. The ASF licenses this file
>>>>>>
>>>>>> # to you under the Apache License, Version 2.0 (the
>>>>>>
>>>>>> # "License"); you may not use this file except in compliance
>>>>>>
>>>>>> # with the License. You may obtain a copy of the License at
>>>>>>
>>>>>> #
>>>>>>
>>>>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>>>>
>>>>>> #
>>>>>>
>>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>>
>>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>>
>>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>>> implied.
>>>>>>
>>>>>> # See the License for the specific language governing permissions and
>>>>>>
>>>>>> # limitations under the License.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> ########### These all have default values as shown
>>>>>>
>>>>>> ########### Additional configuration goes into storm.yaml
>>>>>>
>>>>>>
>>>>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>>>>> */home/ec2-user*"
>>>>>>
>>>>>>
>>>>>>> ### storm.* configs are general configurations
>>>>>>
>>>>>> # the local dir is where jars are kept
>>>>>>
>>>>>> storm.local.dir: "storm-local"
>>>>>>
>>>>>> storm.zookeeper.servers:
>>>>>>
>>>>>> - "*54.68.149.181*"
>>>>>>
>>>>>> storm.zookeeper.port: 2181
>>>>>>
>>>>>> storm.zookeeper.root: "/storm"
>>>>>>
>>>>>> storm.zookeeper.session.timeout: 20000
>>>>>>
>>>>>> storm.zookeeper.connection.timeout: 15000
>>>>>>
>>>>>> storm.zookeeper.retry.times: 5
>>>>>>
>>>>>> storm.zookeeper.retry.interval: 1000
>>>>>>
>>>>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>>>>
>>>>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>>>>
>>>>>> storm.local.mode.zmq: false
>>>>>>
>>>>>> storm.thrift.transport:
>>>>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>>>>
>>>>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>>>>
>>>>>>
>>>>>>> ### nimbus.* configs are for the master
>>>>>>
>>>>>> nimbus.host: "*54.68.149.181*"
>>>>>>
>>>>>> nimbus.thrift.port: 6627
>>>>>>
>>>>>> nimbus.childopts: "-Xmx1024m"
>>>>>>
>>>>>> nimbus.task.timeout.secs: 30
>>>>>>
>>>>>> nimbus.supervisor.timeout.secs: 60
>>>>>>
>>>>>> nimbus.monitor.freq.secs: 10
>>>>>>
>>>>>> nimbus.cleanup.inbox.freq.secs: 600
>>>>>>
>>>>>> nimbus.inbox.jar.expiration.secs: 3600
>>>>>>
>>>>>> nimbus.task.launch.secs: 120
>>>>>>
>>>>>> nimbus.reassign: true
>>>>>>
>>>>>> nimbus.file.copy.expiration.secs: 600
>>>>>>
>>>>>> nimbus.topology.validator:
>>>>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>>>>
>>>>>>
>>>>>>> ### ui.* configs are for the master
>>>>>>
>>>>>> ui.port: 8080
>>>>>>
>>>>>> ui.childopts: "-Xmx768m"
>>>>>>
>>>>>>
>>>>>>> logviewer.port: 8000
>>>>>>
>>>>>> logviewer.childopts: "-Xmx128m"
>>>>>>
>>>>>> logviewer.appender.name: "A1"
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> drpc.port: 3772
>>>>>>
>>>>>> drpc.worker.threads: 64
>>>>>>
>>>>>> drpc.queue.size: 128
>>>>>>
>>>>>> drpc.invocations.port: 3773
>>>>>>
>>>>>> drpc.request.timeout.secs: 600
>>>>>>
>>>>>> drpc.childopts: "-Xmx768m"
>>>>>>
>>>>>>
>>>>>>> transactional.zookeeper.root: "/transactional"
>>>>>>
>>>>>> transactional.zookeeper.servers: null
>>>>>>
>>>>>> transactional.zookeeper.port: null
>>>>>>
>>>>>>
>>>>>>> ### supervisor.* configs are for node supervisors
>>>>>>
>>>>>> # Define the amount of workers that can be run on this machine. Each
>>>>>>> worker is assigned a port to use for communication
>>>>>>
>>>>>> supervisor.slots.ports:
>>>>>>
>>>>>> - 6700
>>>>>>
>>>>>> - 6701
>>>>>>
>>>>>> - 6702
>>>>>>
>>>>>> - 6703
>>>>>>
>>>>>> supervisor.childopts: "-Xmx256m"
>>>>>>
>>>>>> #how long supervisor will wait to ensure that a worker process is
>>>>>>> started
>>>>>>
>>>>>> supervisor.worker.start.timeout.secs: 120
>>>>>>
>>>>>> #how long between heartbeats until supervisor considers that worker
>>>>>>> dead and tries to restart it
>>>>>>
>>>>>> supervisor.worker.timeout.secs: 30
>>>>>>
>>>>>> #how frequently the supervisor checks on the status of the processes
>>>>>>> it's monitoring and restarts if necessary
>>>>>>
>>>>>> supervisor.monitor.frequency.secs: 3
>>>>>>
>>>>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>>>>> nimbus)
>>>>>>
>>>>>> supervisor.heartbeat.frequency.secs: 5
>>>>>>
>>>>>> supervisor.enable: true
>>>>>>
>>>>>>
>>>>>>> ### worker.* configs are for task workers
>>>>>>
>>>>>> worker.childopts: "-Xmx768m"
>>>>>>
>>>>>> worker.heartbeat.frequency.secs: 1
>>>>>>
>>>>>>
>>>>>>> task.heartbeat.frequency.secs: 3
>>>>>>
>>>>>> task.refresh.poll.secs: 10
>>>>>>
>>>>>>
>>>>>>> zmq.threads: 1
>>>>>>
>>>>>> zmq.linger.millis: 5000
>>>>>>
>>>>>> zmq.hwm: 0
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> storm.messaging.netty.server_worker_threads: 1
>>>>>>
>>>>>> storm.messaging.netty.client_worker_threads: 1
>>>>>>
>>>>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>>>>
>>>>>> storm.messaging.netty.max_retries: 30
>>>>>>
>>>>>> storm.messaging.netty.max_wait_ms: 1000
>>>>>>
>>>>>> storm.messaging.netty.min_wait_ms: 100
>>>>>>
>>>>>>
>>>>>>> ### topology.* configs are for specific executing storms
>>>>>>
>>>>>> topology.enable.message.timeouts: true
>>>>>>
>>>>>> topology.debug: false
>>>>>>
>>>>>> topology.optimize: true
>>>>>>
>>>>>> topology.workers: 1
>>>>>>
>>>>>> topology.acker.executors: null
>>>>>>
>>>>>> topology.tasks: null
>>>>>>
>>>>>> # maximum amount of time a message has to complete before it's
>>>>>>> considered failed
>>>>>>
>>>>>> topology.message.timeout.secs: 30
>>>>>>
>>>>>> topology.skip.missing.kryo.registrations: false
>>>>>>
>>>>>> topology.max.task.parallelism: null
>>>>>>
>>>>>> topology.max.spout.pending: null
>>>>>>
>>>>>> topology.state.synchronization.timeout.secs: 60
>>>>>>
>>>>>> topology.stats.sample.rate: 0.05
>>>>>>
>>>>>> topology.builtin.metrics.bucket.size.secs: 60
>>>>>>
>>>>>> topology.fall.back.on.java.serialization: true
>>>>>>
>>>>>> topology.worker.childopts: null
>>>>>>
>>>>>> topology.executor.receive.buffer.size: 1024 #batched
>>>>>>
>>>>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>>>>
>>>>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot
>>>>>>> of problems (heartbeat thread gets starved, throughput plummets)
>>>>>>
>>>>>> topology.transfer.buffer.size: 1024 # batched
>>>>>>
>>>>>> topology.tick.tuple.freq.secs: null
>>>>>>
>>>>>> topology.worker.shared.thread.pool.size: 4
>>>>>>
>>>>>> topology.disruptor.wait.strategy:
>>>>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>>>>
>>>>>> topology.spout.wait.strategy:
>>>>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>>>>
>>>>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>>>>
>>>>>> topology.error.throttle.interval.secs: 10
>>>>>>
>>>>>> topology.max.error.report.per.interval: 5
>>>>>>
>>>>>> topology.kryo.factory:
>>>>>>> "backtype.storm.serialization.DefaultKryoFactory"
>>>>>>
>>>>>> topology.tuple.serializer:
>>>>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>>>>
>>>>>> topology.trident.batch.emit.interval.millis: 500
>>>>>>
>>>>>>
>>>>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>>>>
>>>>>>
>>>>>
>>>>> bin/storm nimbus &
>>>>>
>>>>> bin/storm supervisor &
>>>>>
>>>>> bin/storm ui &
>>>>>
>>>>> *I launch another basic EC2 CentOS machine and do the exact same
>>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>>> nimbus &" at the end*
>>>>>
>>>>> Any thoughts would be greatly appreciated. Thanks so much for all of
>>>>> your help. I'm sure someone else has done this before!
>>>>>
>>>>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
>>>>> wrote:
>>>>>
>>>>>> Although, implementing the Storm cluster manually would be really
>>>>>> nice learning, I would suggest using HortonWorks distribution which comes
>>>>>> with Storm as OOTB solution and you can configure everything from Ambari
>>>>>> UI. We are using Storm on Amazon EC2 machine, though it is right now in
>>>>>> beta stage. We are going to move to production in coming 2-3 months.
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>
>>>>>>> You should be able to see the error it gives by going to the web
>>>>>>> page yourself at: http://54.68.149.181:8080. I am only using this
>>>>>>> account to test and see if I can even get storm to work, so these machines
>>>>>>> are only for testing. Perhaps someone could tell me what the storm.yaml
>>>>>>> file should look like for this setup?
>>>>>>>
>>>>>>> -Thanks, Stephne
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>
>>>>>>>> I'm getting kind of confused by the storm.yaml file. Should I be
>>>>>>>> using the default.yaml and just modify the zookeeper and nimbus ip, or
>>>>>>>> should I use a bran new storm.yaml?
>>>>>>>>
>>>>>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper
>>>>>>>> is on the nimbus machine. what should the storm.yaml look like on my worker
>>>>>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>>>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>>>>>> very basic cluster up and running.
>>>>>>>>
>>>>>>>> -Thanks, Stephen
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>>>>> nothing got it working for me!
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>>>>
>>>>>>>>>> The worker connects to the thrift port and not the ui port. You
>>>>>>>>>> need to open port 6627 or whatever is the value being set in storm.yaml
>>>>>>>>>> using property “nimbus.thrift.port”.
>>>>>>>>>>
>>>>>>>>>> Based on the configuration that you have pointed so far it seems
>>>>>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Parth
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>
>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>>>>> machine is restarted)*
>>>>>>>>>>
>>>>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip
>>>>>>>>>> not 127.0.0.1.
>>>>>>>>>>
>>>>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>>>>> 54.68.149.181"
>>>>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>>>>> access the UI using it?
>>>>>>>>>> -Harsha
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>>>>
>>>>>>>>>> Harsha,
>>>>>>>>>>
>>>>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>>>>
>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>> - "127.0.0.1"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>>>>
>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>>>>
>>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>>> - "54.68.149.181"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>>>>
>>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>>
>>>>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>>>>> host.
>>>>>>>>>> -Thanks, Stephen
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> There aren't any errors in worker machine supervisor logs. Are
>>>>>>>>>> you using the same storm.yaml for both the machines and also are you able
>>>>>>>>>> to ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>>>>> -Harsha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>>>>
>>>>>>>>>> Harsha,
>>>>>>>>>>
>>>>>>>>>> Thanks so much for getting back with me. I will check the logs,
>>>>>>>>>> but I don't seem to get any error messages. I have a nimbus AWS machine
>>>>>>>>>> with zookeeper on it and a worker AWS machine.
>>>>>>>>>>
>>>>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>>>>
>>>>>>>>>> bin/storm nimbus &
>>>>>>>>>> bin/storm supervisor &
>>>>>>>>>> bin/storm ui
>>>>>>>>>>
>>>>>>>>>> On the worker machine I run:
>>>>>>>>>> bin/storm supervisor
>>>>>>>>>>
>>>>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>>>>>>> the nimbus machine.
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Stephen,
>>>>>>>>>> What are the issues you are seeing.
>>>>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in
>>>>>>>>>> the storm configuration file"
>>>>>>>>>>
>>>>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are
>>>>>>>>>> able to connect to your zookeeper cluster.
>>>>>>>>>> Check your logs under storm_inst/logs/ for any errors when you
>>>>>>>>>> try to start nimbus or supervisors.
>>>>>>>>>> If you are installing it manually try following these steps if
>>>>>>>>>> you are not already done.
>>>>>>>>>>
>>>>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>>>>> -Harsha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>>>>
>>>>>>>>>> All,
>>>>>>>>>>
>>>>>>>>>> I would greatly appreciate any help that anyone would afford.
>>>>>>>>>> I've been trying to setup a storm cluster on AWS for a few weeks now on
>>>>>>>>>> centOS EC2 machines. So far, I haven't been able to get a cluster built. I
>>>>>>>>>> can get a supervisor and nimbus to run on a single machine, but I can't
>>>>>>>>>> figure out how to get another worker to connect to nimbus. How do worker
>>>>>>>>>> machines "know" how to connect to nimbus? Is it in the storm configuration
>>>>>>>>>> file? I've gone through many tutorials and the official documentation, but
>>>>>>>>>> this point doesn't seem to be covered anywhere in sufficient detail for a
>>>>>>>>>> new guy like me.
>>>>>>>>>>
>>>>>>>>>> Some of you may be tempted to point me toward storm-deploy, but
>>>>>>>>>> I spent four days trying to get that to work until I gave up. I'm having
>>>>>>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>>>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>>>>>>> offer me any inputs you can!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Vikas Agarwal
>>>>>> 91 – 9928301411
>>>>>>
>>>>>> InfoObjects, Inc.
>>>>>> Execution Matters
>>>>>> http://www.infoobjects.com
>>>>>> 2041 Mission College Boulevard, #280
>>>>>> Santa Clara, CA 95054
>>>>>> +1 (408) 988-2000 Work
>>>>>> +1 (408) 716-2726 Fax
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Vikas Agarwal
>>>> 91 – 9928301411
>>>>
>>>> InfoObjects, Inc.
>>>> Execution Matters
>>>> http://www.infoobjects.com
>>>> 2041 Mission College Boulevard, #280
>>>> Santa Clara, CA 95054
>>>> +1 (408) 988-2000 Work
>>>> +1 (408) 716-2726 Fax
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
HA HA HA HA!!!! Vikas, you crazy genius!!!!!!!!!!! It works!!!! It
works!!!!!!! :D :D :D :D. How much beer do I owe you? After hours and hours
and hours of work... it was a port issue... Thank you so much everyone for
your help and input!
On Tue, Sep 9, 2014 at 8:27 AM, Vikas Agarwal <vi...@infoobjects.com> wrote:
> Did you open following ports in Ec2 security group:
>
> 2181
> 3772
> 3773
> 6627
> 6700
> 6701
> 6702
> 6703
>
> Some of the ports might not be required to be opened, but I guess try
> opening these ports for the same security group not for the world and start
> second instance within same security group.
>
>
> On Tue, Sep 9, 2014 at 5:46 PM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> My apologies, I run "bin/storm supervisor &" at the end. That was a bad
>> copy and paste.
>>
>> On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com>
>> wrote:
>>
>>> >>*I launch another basic EC2 CentOS machine and do the exact same
>>> commands except that I don't install zookeeper and I only run "bin/storm
>>> nimbus &" at the end*<<
>>>
>>> Why would you run nimbus on two machines?
>>>
>>> Yes, by default Hortonworks provides one supervisor node par machine, so
>>> I am trying to add new EC2 machine with supervisor role. And it worked upto
>>> some extent but still not fully functional. I am still testing its linear
>>> scalability.
>>>
>>>
>>> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
>>> hartzell.stephen@gmail.com> wrote:
>>>
>>>> Vikas,
>>>>
>>>>
>>>> I've tried to use the HortonWorks distribution, but that only
>>>> provides one supervisor and nimbus on one virtual machine. I'm excited to
>>>> hear that you have storm working on AWS EC2 machines because that is
>>>> exactly what I am trying to do! Right now we're still in the development
>>>> stage, so all we are trying to do is to have one worker machine connect to
>>>> one nimbus machine. So far we haven't got this work.
>>>>
>>>> Although it might be lengthy, let me go ahead and post the commands
>>>> I'm using to setup the nimbus machine.
>>>>
>>>>
>>>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to
>>>> TCP connections (it's public IP is 54.68.149.181)*
>>>>
>>>> sudo yum update
>>>>
>>>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>>>
>>>> *# Install zeromq*
>>>>
>>>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>>>
>>>> tar –zxvf zeromq-2.1.7.tar.gz
>>>>
>>>> cd zeromq-2.1.7.tar.gz
>>>>
>>>> ./configure
>>>>
>>>> make
>>>>
>>>> sudo make install
>>>>
>>>> cd ~
>>>>
>>>> *# Install jzmq*
>>>>
>>>> sudo yum install git java-devel libtool
>>>>
>>>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>>>> JDK required for jzmq in the configuration stage*
>>>>
>>>> cd $JAVA_HOME
>>>>
>>>> sudo ln –s include include *#
>>>> JDK include directory has headers required for jzmq*
>>>>
>>>> cd ~
>>>>
>>>> git clone https://github.com/nathanmarz/jzmq.git
>>>>
>>>> cd jzmq/src/
>>>>
>>>> CLASSPATH=.:./.:$CLASSPATH
>>>>
>>>> touch classdist_noinst.stamp
>>>>
>>>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>>>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>>>> org/zeromq/ZMQStreamer.java
>>>>
>>>> cd ~/jzmq
>>>>
>>>> ./autogen.sh
>>>>
>>>> ./configure
>>>>
>>>> make
>>>>
>>>> sudo make install
>>>>
>>>> cd ~
>>>>
>>>> *# Download zookeeper*
>>>>
>>>> wget
>>>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>>>
>>>> tar -zxvf zookeeper-3.4.6.tar.gz
>>>>
>>>> cd zookeeper-3.4.6
>>>>
>>>> vi conf/zoo.cfg
>>>>
>>>>
>>>> tickTime=2000
>>>>
>>>> dataDir=/tmp/zookeeper
>>>>
>>>> clientPort=2181
>>>>
>>>>
>>>> mkdir /tmp/zookeeper
>>>> bin/zkServer.sh start
>>>> bin/zkCli.sh -server 54.68.149.181:2181
>>>>
>>>>
>>>> *# Download storm and modify configuration file (changes over
>>>> default.yaml shown in bold)*
>>>>
>>>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>>>> storm-0.8.2.zip
>>>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>>>
>>>> cd storm-0.8.2.zip
>>>>
>>>> vi conf/storm.yaml
>>>>
>>>>
>>>> # Licensed to the Apache Software Foundation (ASF) under one
>>>>>
>>>>> # or more contributor license agreements. See the NOTICE file
>>>>>
>>>>> # distributed with this work for additional information
>>>>>
>>>>> # regarding copyright ownership. The ASF licenses this file
>>>>>
>>>>> # to you under the Apache License, Version 2.0 (the
>>>>>
>>>>> # "License"); you may not use this file except in compliance
>>>>>
>>>>> # with the License. You may obtain a copy of the License at
>>>>>
>>>>> #
>>>>>
>>>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>>>
>>>>> #
>>>>>
>>>>> # Unless required by applicable law or agreed to in writing, software
>>>>>
>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>>
>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>>> implied.
>>>>>
>>>>> # See the License for the specific language governing permissions and
>>>>>
>>>>> # limitations under the License.
>>>>>
>>>>>
>>>>>>
>>>>>> ########### These all have default values as shown
>>>>>
>>>>> ########### Additional configuration goes into storm.yaml
>>>>>
>>>>>
>>>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>>>> */home/ec2-user*"
>>>>>
>>>>>
>>>>>> ### storm.* configs are general configurations
>>>>>
>>>>> # the local dir is where jars are kept
>>>>>
>>>>> storm.local.dir: "storm-local"
>>>>>
>>>>> storm.zookeeper.servers:
>>>>>
>>>>> - "*54.68.149.181*"
>>>>>
>>>>> storm.zookeeper.port: 2181
>>>>>
>>>>> storm.zookeeper.root: "/storm"
>>>>>
>>>>> storm.zookeeper.session.timeout: 20000
>>>>>
>>>>> storm.zookeeper.connection.timeout: 15000
>>>>>
>>>>> storm.zookeeper.retry.times: 5
>>>>>
>>>>> storm.zookeeper.retry.interval: 1000
>>>>>
>>>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>>>
>>>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>>>
>>>>> storm.local.mode.zmq: false
>>>>>
>>>>> storm.thrift.transport:
>>>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>>>
>>>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>>>
>>>>>
>>>>>> ### nimbus.* configs are for the master
>>>>>
>>>>> nimbus.host: "*54.68.149.181*"
>>>>>
>>>>> nimbus.thrift.port: 6627
>>>>>
>>>>> nimbus.childopts: "-Xmx1024m"
>>>>>
>>>>> nimbus.task.timeout.secs: 30
>>>>>
>>>>> nimbus.supervisor.timeout.secs: 60
>>>>>
>>>>> nimbus.monitor.freq.secs: 10
>>>>>
>>>>> nimbus.cleanup.inbox.freq.secs: 600
>>>>>
>>>>> nimbus.inbox.jar.expiration.secs: 3600
>>>>>
>>>>> nimbus.task.launch.secs: 120
>>>>>
>>>>> nimbus.reassign: true
>>>>>
>>>>> nimbus.file.copy.expiration.secs: 600
>>>>>
>>>>> nimbus.topology.validator:
>>>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>>>
>>>>>
>>>>>> ### ui.* configs are for the master
>>>>>
>>>>> ui.port: 8080
>>>>>
>>>>> ui.childopts: "-Xmx768m"
>>>>>
>>>>>
>>>>>> logviewer.port: 8000
>>>>>
>>>>> logviewer.childopts: "-Xmx128m"
>>>>>
>>>>> logviewer.appender.name: "A1"
>>>>>
>>>>>
>>>>>>
>>>>>> drpc.port: 3772
>>>>>
>>>>> drpc.worker.threads: 64
>>>>>
>>>>> drpc.queue.size: 128
>>>>>
>>>>> drpc.invocations.port: 3773
>>>>>
>>>>> drpc.request.timeout.secs: 600
>>>>>
>>>>> drpc.childopts: "-Xmx768m"
>>>>>
>>>>>
>>>>>> transactional.zookeeper.root: "/transactional"
>>>>>
>>>>> transactional.zookeeper.servers: null
>>>>>
>>>>> transactional.zookeeper.port: null
>>>>>
>>>>>
>>>>>> ### supervisor.* configs are for node supervisors
>>>>>
>>>>> # Define the amount of workers that can be run on this machine. Each
>>>>>> worker is assigned a port to use for communication
>>>>>
>>>>> supervisor.slots.ports:
>>>>>
>>>>> - 6700
>>>>>
>>>>> - 6701
>>>>>
>>>>> - 6702
>>>>>
>>>>> - 6703
>>>>>
>>>>> supervisor.childopts: "-Xmx256m"
>>>>>
>>>>> #how long supervisor will wait to ensure that a worker process is
>>>>>> started
>>>>>
>>>>> supervisor.worker.start.timeout.secs: 120
>>>>>
>>>>> #how long between heartbeats until supervisor considers that worker
>>>>>> dead and tries to restart it
>>>>>
>>>>> supervisor.worker.timeout.secs: 30
>>>>>
>>>>> #how frequently the supervisor checks on the status of the processes
>>>>>> it's monitoring and restarts if necessary
>>>>>
>>>>> supervisor.monitor.frequency.secs: 3
>>>>>
>>>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>>>> nimbus)
>>>>>
>>>>> supervisor.heartbeat.frequency.secs: 5
>>>>>
>>>>> supervisor.enable: true
>>>>>
>>>>>
>>>>>> ### worker.* configs are for task workers
>>>>>
>>>>> worker.childopts: "-Xmx768m"
>>>>>
>>>>> worker.heartbeat.frequency.secs: 1
>>>>>
>>>>>
>>>>>> task.heartbeat.frequency.secs: 3
>>>>>
>>>>> task.refresh.poll.secs: 10
>>>>>
>>>>>
>>>>>> zmq.threads: 1
>>>>>
>>>>> zmq.linger.millis: 5000
>>>>>
>>>>> zmq.hwm: 0
>>>>>
>>>>>
>>>>>>
>>>>>> storm.messaging.netty.server_worker_threads: 1
>>>>>
>>>>> storm.messaging.netty.client_worker_threads: 1
>>>>>
>>>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>>>
>>>>> storm.messaging.netty.max_retries: 30
>>>>>
>>>>> storm.messaging.netty.max_wait_ms: 1000
>>>>>
>>>>> storm.messaging.netty.min_wait_ms: 100
>>>>>
>>>>>
>>>>>> ### topology.* configs are for specific executing storms
>>>>>
>>>>> topology.enable.message.timeouts: true
>>>>>
>>>>> topology.debug: false
>>>>>
>>>>> topology.optimize: true
>>>>>
>>>>> topology.workers: 1
>>>>>
>>>>> topology.acker.executors: null
>>>>>
>>>>> topology.tasks: null
>>>>>
>>>>> # maximum amount of time a message has to complete before it's
>>>>>> considered failed
>>>>>
>>>>> topology.message.timeout.secs: 30
>>>>>
>>>>> topology.skip.missing.kryo.registrations: false
>>>>>
>>>>> topology.max.task.parallelism: null
>>>>>
>>>>> topology.max.spout.pending: null
>>>>>
>>>>> topology.state.synchronization.timeout.secs: 60
>>>>>
>>>>> topology.stats.sample.rate: 0.05
>>>>>
>>>>> topology.builtin.metrics.bucket.size.secs: 60
>>>>>
>>>>> topology.fall.back.on.java.serialization: true
>>>>>
>>>>> topology.worker.childopts: null
>>>>>
>>>>> topology.executor.receive.buffer.size: 1024 #batched
>>>>>
>>>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>>>
>>>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot of
>>>>>> problems (heartbeat thread gets starved, throughput plummets)
>>>>>
>>>>> topology.transfer.buffer.size: 1024 # batched
>>>>>
>>>>> topology.tick.tuple.freq.secs: null
>>>>>
>>>>> topology.worker.shared.thread.pool.size: 4
>>>>>
>>>>> topology.disruptor.wait.strategy:
>>>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>>>
>>>>> topology.spout.wait.strategy:
>>>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>>>
>>>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>>>
>>>>> topology.error.throttle.interval.secs: 10
>>>>>
>>>>> topology.max.error.report.per.interval: 5
>>>>>
>>>>> topology.kryo.factory:
>>>>>> "backtype.storm.serialization.DefaultKryoFactory"
>>>>>
>>>>> topology.tuple.serializer:
>>>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>>>
>>>>> topology.trident.batch.emit.interval.millis: 500
>>>>>
>>>>>
>>>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>>>
>>>>>
>>>>
>>>> bin/storm nimbus &
>>>>
>>>> bin/storm supervisor &
>>>>
>>>> bin/storm ui &
>>>>
>>>> *I launch another basic EC2 CentOS machine and do the exact same
>>>> commands except that I don't install zookeeper and I only run "bin/storm
>>>> nimbus &" at the end*
>>>>
>>>> Any thoughts would be greatly appreciated. Thanks so much for all of
>>>> your help. I'm sure someone else has done this before!
>>>>
>>>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
>>>> wrote:
>>>>
>>>>> Although, implementing the Storm cluster manually would be really nice
>>>>> learning, I would suggest using HortonWorks distribution which comes with
>>>>> Storm as OOTB solution and you can configure everything from Ambari UI. We
>>>>> are using Storm on Amazon EC2 machine, though it is right now in beta
>>>>> stage. We are going to move to production in coming 2-3 months.
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>
>>>>>> All,
>>>>>>
>>>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>>>> java.net.ConnectException: Connection refused
>>>>>>
>>>>>> You should be able to see the error it gives by going to the web page
>>>>>> yourself at: http://54.68.149.181:8080. I am only using this account
>>>>>> to test and see if I can even get storm to work, so these machines are only
>>>>>> for testing. Perhaps someone could tell me what the storm.yaml file should
>>>>>> look like for this setup?
>>>>>>
>>>>>> -Thanks, Stephne
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>
>>>>>>> I'm getting kind of confused by the storm.yaml file. Should I be
>>>>>>> using the default.yaml and just modify the zookeeper and nimbus ip, or
>>>>>>> should I use a bran new storm.yaml?
>>>>>>>
>>>>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper
>>>>>>> is on the nimbus machine. what should the storm.yaml look like on my worker
>>>>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>>>>> very basic cluster up and running.
>>>>>>>
>>>>>>> -Thanks, Stephen
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>>
>>>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>>>> nothing got it working for me!
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>>>
>>>>>>>>> The worker connects to the thrift port and not the ui port. You
>>>>>>>>> need to open port 6627 or whatever is the value being set in storm.yaml
>>>>>>>>> using property “nimbus.thrift.port”.
>>>>>>>>>
>>>>>>>>> Based on the configuration that you have pointed so far it seems
>>>>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Parth
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>
>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>> - "127.0.0.1"
>>>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>>>> machine is restarted)*
>>>>>>>>>
>>>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip
>>>>>>>>> not 127.0.0.1.
>>>>>>>>>
>>>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>>>> 54.68.149.181"
>>>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>>>> access the UI using it?
>>>>>>>>> -Harsha
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>>>
>>>>>>>>> Harsha,
>>>>>>>>>
>>>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>>>
>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>> - "127.0.0.1"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>>>
>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>>>
>>>>>>>>> storm.zookeeper.servers:
>>>>>>>>> - "54.68.149.181"
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>>>
>>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>>
>>>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>>>> host.
>>>>>>>>> -Thanks, Stephen
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>>>>>>> using the same storm.yaml for both the machines and also are you able to
>>>>>>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>>>> -Harsha
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>>>
>>>>>>>>> Harsha,
>>>>>>>>>
>>>>>>>>> Thanks so much for getting back with me. I will check the logs,
>>>>>>>>> but I don't seem to get any error messages. I have a nimbus AWS machine
>>>>>>>>> with zookeeper on it and a worker AWS machine.
>>>>>>>>>
>>>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>>>
>>>>>>>>> bin/storm nimbus &
>>>>>>>>> bin/storm supervisor &
>>>>>>>>> bin/storm ui
>>>>>>>>>
>>>>>>>>> On the worker machine I run:
>>>>>>>>> bin/storm supervisor
>>>>>>>>>
>>>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>>>>>> the nimbus machine.
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Stephen,
>>>>>>>>> What are the issues you are seeing.
>>>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in
>>>>>>>>> the storm configuration file"
>>>>>>>>>
>>>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able
>>>>>>>>> to connect to your zookeeper cluster.
>>>>>>>>> Check your logs under storm_inst/logs/ for any errors when you try
>>>>>>>>> to start nimbus or supervisors.
>>>>>>>>> If you are installing it manually try following these steps if you
>>>>>>>>> are not already done.
>>>>>>>>>
>>>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>>>> -Harsha
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>>>
>>>>>>>>> All,
>>>>>>>>>
>>>>>>>>> I would greatly appreciate any help that anyone would afford. I've
>>>>>>>>> been trying to setup a storm cluster on AWS for a few weeks now on centOS
>>>>>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can get
>>>>>>>>> a supervisor and nimbus to run on a single machine, but I can't figure out
>>>>>>>>> how to get another worker to connect to nimbus. How do worker machines
>>>>>>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>>>>>>> I've gone through many tutorials and the official documentation, but this
>>>>>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>>>>>>> guy like me.
>>>>>>>>>
>>>>>>>>> Some of you may be tempted to point me toward storm-deploy, but
>>>>>>>>> I spent four days trying to get that to work until I gave up. I'm having
>>>>>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>>>>>> offer me any inputs you can!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Vikas Agarwal
>>>>> 91 – 9928301411
>>>>>
>>>>> InfoObjects, Inc.
>>>>> Execution Matters
>>>>> http://www.infoobjects.com
>>>>> 2041 Mission College Boulevard, #280
>>>>> Santa Clara, CA 95054
>>>>> +1 (408) 988-2000 Work
>>>>> +1 (408) 716-2726 Fax
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Vikas Agarwal
>>> 91 – 9928301411
>>>
>>> InfoObjects, Inc.
>>> Execution Matters
>>> http://www.infoobjects.com
>>> 2041 Mission College Boulevard, #280
>>> Santa Clara, CA 95054
>>> +1 (408) 988-2000 Work
>>> +1 (408) 716-2726 Fax
>>>
>>>
>>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: How Do Workers Connect To Nimbus
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Did you open following ports in Ec2 security group:
2181
3772
3773
6627
6700
6701
6702
6703
Some of the ports might not be required to be opened, but I guess try
opening these ports for the same security group not for the world and start
second instance within same security group.
On Tue, Sep 9, 2014 at 5:46 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> My apologies, I run "bin/storm supervisor &" at the end. That was a bad
> copy and paste.
>
> On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> >>*I launch another basic EC2 CentOS machine and do the exact same
>> commands except that I don't install zookeeper and I only run "bin/storm
>> nimbus &" at the end*<<
>>
>> Why would you run nimbus on two machines?
>>
>> Yes, by default Hortonworks provides one supervisor node par machine, so
>> I am trying to add new EC2 machine with supervisor role. And it worked upto
>> some extent but still not fully functional. I am still testing its linear
>> scalability.
>>
>>
>> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> Vikas,
>>>
>>>
>>> I've tried to use the HortonWorks distribution, but that only provides
>>> one supervisor and nimbus on one virtual machine. I'm excited to hear that
>>> you have storm working on AWS EC2 machines because that is exactly what I
>>> am trying to do! Right now we're still in the development stage, so all we
>>> are trying to do is to have one worker machine connect to one nimbus
>>> machine. So far we haven't got this work.
>>>
>>> Although it might be lengthy, let me go ahead and post the commands
>>> I'm using to setup the nimbus machine.
>>>
>>>
>>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to
>>> TCP connections (it's public IP is 54.68.149.181)*
>>>
>>> sudo yum update
>>>
>>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>>
>>> *# Install zeromq*
>>>
>>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>>
>>> tar –zxvf zeromq-2.1.7.tar.gz
>>>
>>> cd zeromq-2.1.7.tar.gz
>>>
>>> ./configure
>>>
>>> make
>>>
>>> sudo make install
>>>
>>> cd ~
>>>
>>> *# Install jzmq*
>>>
>>> sudo yum install git java-devel libtool
>>>
>>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>>> JDK required for jzmq in the configuration stage*
>>>
>>> cd $JAVA_HOME
>>>
>>> sudo ln –s include include *#
>>> JDK include directory has headers required for jzmq*
>>>
>>> cd ~
>>>
>>> git clone https://github.com/nathanmarz/jzmq.git
>>>
>>> cd jzmq/src/
>>>
>>> CLASSPATH=.:./.:$CLASSPATH
>>>
>>> touch classdist_noinst.stamp
>>>
>>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>>> org/zeromq/ZMQStreamer.java
>>>
>>> cd ~/jzmq
>>>
>>> ./autogen.sh
>>>
>>> ./configure
>>>
>>> make
>>>
>>> sudo make install
>>>
>>> cd ~
>>>
>>> *# Download zookeeper*
>>>
>>> wget
>>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>>
>>> tar -zxvf zookeeper-3.4.6.tar.gz
>>>
>>> cd zookeeper-3.4.6
>>>
>>> vi conf/zoo.cfg
>>>
>>>
>>> tickTime=2000
>>>
>>> dataDir=/tmp/zookeeper
>>>
>>> clientPort=2181
>>>
>>>
>>> mkdir /tmp/zookeeper
>>> bin/zkServer.sh start
>>> bin/zkCli.sh -server 54.68.149.181:2181
>>>
>>>
>>> *# Download storm and modify configuration file (changes over
>>> default.yaml shown in bold)*
>>>
>>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>>> storm-0.8.2.zip
>>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>>
>>> cd storm-0.8.2.zip
>>>
>>> vi conf/storm.yaml
>>>
>>>
>>> # Licensed to the Apache Software Foundation (ASF) under one
>>>>
>>>> # or more contributor license agreements. See the NOTICE file
>>>>
>>>> # distributed with this work for additional information
>>>>
>>>> # regarding copyright ownership. The ASF licenses this file
>>>>
>>>> # to you under the Apache License, Version 2.0 (the
>>>>
>>>> # "License"); you may not use this file except in compliance
>>>>
>>>> # with the License. You may obtain a copy of the License at
>>>>
>>>> #
>>>>
>>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>>
>>>> #
>>>>
>>>> # Unless required by applicable law or agreed to in writing, software
>>>>
>>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>>
>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>>> implied.
>>>>
>>>> # See the License for the specific language governing permissions and
>>>>
>>>> # limitations under the License.
>>>>
>>>>
>>>>>
>>>>> ########### These all have default values as shown
>>>>
>>>> ########### Additional configuration goes into storm.yaml
>>>>
>>>>
>>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>>> */home/ec2-user*"
>>>>
>>>>
>>>>> ### storm.* configs are general configurations
>>>>
>>>> # the local dir is where jars are kept
>>>>
>>>> storm.local.dir: "storm-local"
>>>>
>>>> storm.zookeeper.servers:
>>>>
>>>> - "*54.68.149.181*"
>>>>
>>>> storm.zookeeper.port: 2181
>>>>
>>>> storm.zookeeper.root: "/storm"
>>>>
>>>> storm.zookeeper.session.timeout: 20000
>>>>
>>>> storm.zookeeper.connection.timeout: 15000
>>>>
>>>> storm.zookeeper.retry.times: 5
>>>>
>>>> storm.zookeeper.retry.interval: 1000
>>>>
>>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>>
>>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>>
>>>> storm.local.mode.zmq: false
>>>>
>>>> storm.thrift.transport:
>>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>>
>>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>>
>>>>
>>>>> ### nimbus.* configs are for the master
>>>>
>>>> nimbus.host: "*54.68.149.181*"
>>>>
>>>> nimbus.thrift.port: 6627
>>>>
>>>> nimbus.childopts: "-Xmx1024m"
>>>>
>>>> nimbus.task.timeout.secs: 30
>>>>
>>>> nimbus.supervisor.timeout.secs: 60
>>>>
>>>> nimbus.monitor.freq.secs: 10
>>>>
>>>> nimbus.cleanup.inbox.freq.secs: 600
>>>>
>>>> nimbus.inbox.jar.expiration.secs: 3600
>>>>
>>>> nimbus.task.launch.secs: 120
>>>>
>>>> nimbus.reassign: true
>>>>
>>>> nimbus.file.copy.expiration.secs: 600
>>>>
>>>> nimbus.topology.validator:
>>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>>
>>>>
>>>>> ### ui.* configs are for the master
>>>>
>>>> ui.port: 8080
>>>>
>>>> ui.childopts: "-Xmx768m"
>>>>
>>>>
>>>>> logviewer.port: 8000
>>>>
>>>> logviewer.childopts: "-Xmx128m"
>>>>
>>>> logviewer.appender.name: "A1"
>>>>
>>>>
>>>>>
>>>>> drpc.port: 3772
>>>>
>>>> drpc.worker.threads: 64
>>>>
>>>> drpc.queue.size: 128
>>>>
>>>> drpc.invocations.port: 3773
>>>>
>>>> drpc.request.timeout.secs: 600
>>>>
>>>> drpc.childopts: "-Xmx768m"
>>>>
>>>>
>>>>> transactional.zookeeper.root: "/transactional"
>>>>
>>>> transactional.zookeeper.servers: null
>>>>
>>>> transactional.zookeeper.port: null
>>>>
>>>>
>>>>> ### supervisor.* configs are for node supervisors
>>>>
>>>> # Define the amount of workers that can be run on this machine. Each
>>>>> worker is assigned a port to use for communication
>>>>
>>>> supervisor.slots.ports:
>>>>
>>>> - 6700
>>>>
>>>> - 6701
>>>>
>>>> - 6702
>>>>
>>>> - 6703
>>>>
>>>> supervisor.childopts: "-Xmx256m"
>>>>
>>>> #how long supervisor will wait to ensure that a worker process is
>>>>> started
>>>>
>>>> supervisor.worker.start.timeout.secs: 120
>>>>
>>>> #how long between heartbeats until supervisor considers that worker
>>>>> dead and tries to restart it
>>>>
>>>> supervisor.worker.timeout.secs: 30
>>>>
>>>> #how frequently the supervisor checks on the status of the processes
>>>>> it's monitoring and restarts if necessary
>>>>
>>>> supervisor.monitor.frequency.secs: 3
>>>>
>>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>>> nimbus)
>>>>
>>>> supervisor.heartbeat.frequency.secs: 5
>>>>
>>>> supervisor.enable: true
>>>>
>>>>
>>>>> ### worker.* configs are for task workers
>>>>
>>>> worker.childopts: "-Xmx768m"
>>>>
>>>> worker.heartbeat.frequency.secs: 1
>>>>
>>>>
>>>>> task.heartbeat.frequency.secs: 3
>>>>
>>>> task.refresh.poll.secs: 10
>>>>
>>>>
>>>>> zmq.threads: 1
>>>>
>>>> zmq.linger.millis: 5000
>>>>
>>>> zmq.hwm: 0
>>>>
>>>>
>>>>>
>>>>> storm.messaging.netty.server_worker_threads: 1
>>>>
>>>> storm.messaging.netty.client_worker_threads: 1
>>>>
>>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>>
>>>> storm.messaging.netty.max_retries: 30
>>>>
>>>> storm.messaging.netty.max_wait_ms: 1000
>>>>
>>>> storm.messaging.netty.min_wait_ms: 100
>>>>
>>>>
>>>>> ### topology.* configs are for specific executing storms
>>>>
>>>> topology.enable.message.timeouts: true
>>>>
>>>> topology.debug: false
>>>>
>>>> topology.optimize: true
>>>>
>>>> topology.workers: 1
>>>>
>>>> topology.acker.executors: null
>>>>
>>>> topology.tasks: null
>>>>
>>>> # maximum amount of time a message has to complete before it's
>>>>> considered failed
>>>>
>>>> topology.message.timeout.secs: 30
>>>>
>>>> topology.skip.missing.kryo.registrations: false
>>>>
>>>> topology.max.task.parallelism: null
>>>>
>>>> topology.max.spout.pending: null
>>>>
>>>> topology.state.synchronization.timeout.secs: 60
>>>>
>>>> topology.stats.sample.rate: 0.05
>>>>
>>>> topology.builtin.metrics.bucket.size.secs: 60
>>>>
>>>> topology.fall.back.on.java.serialization: true
>>>>
>>>> topology.worker.childopts: null
>>>>
>>>> topology.executor.receive.buffer.size: 1024 #batched
>>>>
>>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>>
>>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot of
>>>>> problems (heartbeat thread gets starved, throughput plummets)
>>>>
>>>> topology.transfer.buffer.size: 1024 # batched
>>>>
>>>> topology.tick.tuple.freq.secs: null
>>>>
>>>> topology.worker.shared.thread.pool.size: 4
>>>>
>>>> topology.disruptor.wait.strategy:
>>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>>
>>>> topology.spout.wait.strategy:
>>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>>
>>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>>
>>>> topology.error.throttle.interval.secs: 10
>>>>
>>>> topology.max.error.report.per.interval: 5
>>>>
>>>> topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
>>>>
>>>> topology.tuple.serializer:
>>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>>
>>>> topology.trident.batch.emit.interval.millis: 500
>>>>
>>>>
>>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>>
>>>>
>>>
>>> bin/storm nimbus &
>>>
>>> bin/storm supervisor &
>>>
>>> bin/storm ui &
>>>
>>> *I launch another basic EC2 CentOS machine and do the exact same
>>> commands except that I don't install zookeeper and I only run "bin/storm
>>> nimbus &" at the end*
>>>
>>> Any thoughts would be greatly appreciated. Thanks so much for all of
>>> your help. I'm sure someone else has done this before!
>>>
>>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
>>> wrote:
>>>
>>>> Although, implementing the Storm cluster manually would be really nice
>>>> learning, I would suggest using HortonWorks distribution which comes with
>>>> Storm as OOTB solution and you can configure everything from Ambari UI. We
>>>> are using Storm on Amazon EC2 machine, though it is right now in beta
>>>> stage. We are going to move to production in coming 2-3 months.
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>>> hartzell.stephen@gmail.com> wrote:
>>>>
>>>>> All,
>>>>>
>>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>>> java.net.ConnectException: Connection refused
>>>>>
>>>>> You should be able to see the error it gives by going to the web page
>>>>> yourself at: http://54.68.149.181:8080. I am only using this account
>>>>> to test and see if I can even get storm to work, so these machines are only
>>>>> for testing. Perhaps someone could tell me what the storm.yaml file should
>>>>> look like for this setup?
>>>>>
>>>>> -Thanks, Stephne
>>>>>
>>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>
>>>>>> I'm getting kind of confused by the storm.yaml file. Should I be
>>>>>> using the default.yaml and just modify the zookeeper and nimbus ip, or
>>>>>> should I use a bran new storm.yaml?
>>>>>>
>>>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is
>>>>>> on the nimbus machine. what should the storm.yaml look like on my worker
>>>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>>>> very basic cluster up and running.
>>>>>>
>>>>>> -Thanks, Stephen
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>>
>>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>>> nothing got it working for me!
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>>
>>>>>>>> The worker connects to the thrift port and not the ui port. You
>>>>>>>> need to open port 6627 or whatever is the value being set in storm.yaml
>>>>>>>> using property “nimbus.thrift.port”.
>>>>>>>>
>>>>>>>> Based on the configuration that you have pointed so far it seems
>>>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Parth
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>
>>>>>>>> storm.zookeeper.servers:
>>>>>>>> - "127.0.0.1"
>>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>>> machine is restarted)*
>>>>>>>>
>>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip
>>>>>>>> not 127.0.0.1.
>>>>>>>>
>>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>>> 54.68.149.181"
>>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>>> access the UI using it?
>>>>>>>> -Harsha
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>>
>>>>>>>> Harsha,
>>>>>>>>
>>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>>
>>>>>>>> storm.zookeeper.servers:
>>>>>>>> - "127.0.0.1"
>>>>>>>>
>>>>>>>>
>>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>>
>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>
>>>>>>>>
>>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>>
>>>>>>>> storm.zookeeper.servers:
>>>>>>>> - "54.68.149.181"
>>>>>>>>
>>>>>>>>
>>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>>
>>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>>
>>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>>> host.
>>>>>>>> -Thanks, Stephen
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>>>>>> using the same storm.yaml for both the machines and also are you able to
>>>>>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>>> -Harsha
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>>
>>>>>>>> Harsha,
>>>>>>>>
>>>>>>>> Thanks so much for getting back with me. I will check the logs,
>>>>>>>> but I don't seem to get any error messages. I have a nimbus AWS machine
>>>>>>>> with zookeeper on it and a worker AWS machine.
>>>>>>>>
>>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>>
>>>>>>>> bin/storm nimbus &
>>>>>>>> bin/storm supervisor &
>>>>>>>> bin/storm ui
>>>>>>>>
>>>>>>>> On the worker machine I run:
>>>>>>>> bin/storm supervisor
>>>>>>>>
>>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>>>>> the nimbus machine.
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Stephen,
>>>>>>>> What are the issues you are seeing.
>>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in
>>>>>>>> the storm configuration file"
>>>>>>>>
>>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able
>>>>>>>> to connect to your zookeeper cluster.
>>>>>>>> Check your logs under storm_inst/logs/ for any errors when you try
>>>>>>>> to start nimbus or supervisors.
>>>>>>>> If you are installing it manually try following these steps if you
>>>>>>>> are not already done.
>>>>>>>>
>>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>>> -Harsha
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>>
>>>>>>>> All,
>>>>>>>>
>>>>>>>> I would greatly appreciate any help that anyone would afford. I've
>>>>>>>> been trying to setup a storm cluster on AWS for a few weeks now on centOS
>>>>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can get
>>>>>>>> a supervisor and nimbus to run on a single machine, but I can't figure out
>>>>>>>> how to get another worker to connect to nimbus. How do worker machines
>>>>>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>>>>>> I've gone through many tutorials and the official documentation, but this
>>>>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>>>>>> guy like me.
>>>>>>>>
>>>>>>>> Some of you may be tempted to point me toward storm-deploy, but I
>>>>>>>> spent four days trying to get that to work until I gave up. I'm having
>>>>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>>>>> offer me any inputs you can!
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>>> you have received this communication in error, please contact the sender
>>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Vikas Agarwal
>>>> 91 – 9928301411
>>>>
>>>> InfoObjects, Inc.
>>>> Execution Matters
>>>> http://www.infoobjects.com
>>>> 2041 Mission College Boulevard, #280
>>>> Santa Clara, CA 95054
>>>> +1 (408) 988-2000 Work
>>>> +1 (408) 716-2726 Fax
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
My apologies, I run "bin/storm supervisor &" at the end. That was a bad
copy and paste.
On Tue, Sep 9, 2014 at 8:10 AM, Vikas Agarwal <vi...@infoobjects.com> wrote:
> >>*I launch another basic EC2 CentOS machine and do the exact same
> commands except that I don't install zookeeper and I only run "bin/storm
> nimbus &" at the end*<<
>
> Why would you run nimbus on two machines?
>
> Yes, by default Hortonworks provides one supervisor node par machine, so I
> am trying to add new EC2 machine with supervisor role. And it worked upto
> some extent but still not fully functional. I am still testing its linear
> scalability.
>
>
> On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> Vikas,
>>
>>
>> I've tried to use the HortonWorks distribution, but that only provides
>> one supervisor and nimbus on one virtual machine. I'm excited to hear that
>> you have storm working on AWS EC2 machines because that is exactly what I
>> am trying to do! Right now we're still in the development stage, so all we
>> are trying to do is to have one worker machine connect to one nimbus
>> machine. So far we haven't got this work.
>>
>> Although it might be lengthy, let me go ahead and post the commands I'm
>> using to setup the nimbus machine.
>>
>>
>> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to TCP
>> connections (it's public IP is 54.68.149.181)*
>>
>> sudo yum update
>>
>> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>>
>> *# Install zeromq*
>>
>> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>>
>> tar –zxvf zeromq-2.1.7.tar.gz
>>
>> cd zeromq-2.1.7.tar.gz
>>
>> ./configure
>>
>> make
>>
>> sudo make install
>>
>> cd ~
>>
>> *# Install jzmq*
>>
>> sudo yum install git java-devel libtool
>>
>> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *#
>> JDK required for jzmq in the configuration stage*
>>
>> cd $JAVA_HOME
>>
>> sudo ln –s include include *#
>> JDK include directory has headers required for jzmq*
>>
>> cd ~
>>
>> git clone https://github.com/nathanmarz/jzmq.git
>>
>> cd jzmq/src/
>>
>> CLASSPATH=.:./.:$CLASSPATH
>>
>> touch classdist_noinst.stamp
>>
>> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
>> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
>> org/zeromq/ZMQStreamer.java
>>
>> cd ~/jzmq
>>
>> ./autogen.sh
>>
>> ./configure
>>
>> make
>>
>> sudo make install
>>
>> cd ~
>>
>> *# Download zookeeper*
>>
>> wget
>> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>>
>> tar -zxvf zookeeper-3.4.6.tar.gz
>>
>> cd zookeeper-3.4.6
>>
>> vi conf/zoo.cfg
>>
>>
>> tickTime=2000
>>
>> dataDir=/tmp/zookeeper
>>
>> clientPort=2181
>>
>>
>> mkdir /tmp/zookeeper
>> bin/zkServer.sh start
>> bin/zkCli.sh -server 54.68.149.181:2181
>>
>>
>> *# Download storm and modify configuration file (changes over
>> default.yaml shown in bold)*
>>
>> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
>> storm-0.8.2.zip
>> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>>
>> cd storm-0.8.2.zip
>>
>> vi conf/storm.yaml
>>
>>
>> # Licensed to the Apache Software Foundation (ASF) under one
>>>
>>> # or more contributor license agreements. See the NOTICE file
>>>
>>> # distributed with this work for additional information
>>>
>>> # regarding copyright ownership. The ASF licenses this file
>>>
>>> # to you under the Apache License, Version 2.0 (the
>>>
>>> # "License"); you may not use this file except in compliance
>>>
>>> # with the License. You may obtain a copy of the License at
>>>
>>> #
>>>
>>> # http:# www.apache.org/licenses/LICENSE-2.0
>>>
>>> #
>>>
>>> # Unless required by applicable law or agreed to in writing, software
>>>
>>> # distributed under the License is distributed on an "AS IS" BASIS,
>>>
>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
>>>> implied.
>>>
>>> # See the License for the specific language governing permissions and
>>>
>>> # limitations under the License.
>>>
>>>
>>>>
>>>> ########### These all have default values as shown
>>>
>>> ########### Additional configuration goes into storm.yaml
>>>
>>>
>>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>>> */home/ec2-user*"
>>>
>>>
>>>> ### storm.* configs are general configurations
>>>
>>> # the local dir is where jars are kept
>>>
>>> storm.local.dir: "storm-local"
>>>
>>> storm.zookeeper.servers:
>>>
>>> - "*54.68.149.181*"
>>>
>>> storm.zookeeper.port: 2181
>>>
>>> storm.zookeeper.root: "/storm"
>>>
>>> storm.zookeeper.session.timeout: 20000
>>>
>>> storm.zookeeper.connection.timeout: 15000
>>>
>>> storm.zookeeper.retry.times: 5
>>>
>>> storm.zookeeper.retry.interval: 1000
>>>
>>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>>
>>> storm.cluster.mode: "distributed" # can be distributed or local
>>>
>>> storm.local.mode.zmq: false
>>>
>>> storm.thrift.transport:
>>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>>
>>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>>
>>>
>>>> ### nimbus.* configs are for the master
>>>
>>> nimbus.host: "*54.68.149.181*"
>>>
>>> nimbus.thrift.port: 6627
>>>
>>> nimbus.childopts: "-Xmx1024m"
>>>
>>> nimbus.task.timeout.secs: 30
>>>
>>> nimbus.supervisor.timeout.secs: 60
>>>
>>> nimbus.monitor.freq.secs: 10
>>>
>>> nimbus.cleanup.inbox.freq.secs: 600
>>>
>>> nimbus.inbox.jar.expiration.secs: 3600
>>>
>>> nimbus.task.launch.secs: 120
>>>
>>> nimbus.reassign: true
>>>
>>> nimbus.file.copy.expiration.secs: 600
>>>
>>> nimbus.topology.validator:
>>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>>
>>>
>>>> ### ui.* configs are for the master
>>>
>>> ui.port: 8080
>>>
>>> ui.childopts: "-Xmx768m"
>>>
>>>
>>>> logviewer.port: 8000
>>>
>>> logviewer.childopts: "-Xmx128m"
>>>
>>> logviewer.appender.name: "A1"
>>>
>>>
>>>>
>>>> drpc.port: 3772
>>>
>>> drpc.worker.threads: 64
>>>
>>> drpc.queue.size: 128
>>>
>>> drpc.invocations.port: 3773
>>>
>>> drpc.request.timeout.secs: 600
>>>
>>> drpc.childopts: "-Xmx768m"
>>>
>>>
>>>> transactional.zookeeper.root: "/transactional"
>>>
>>> transactional.zookeeper.servers: null
>>>
>>> transactional.zookeeper.port: null
>>>
>>>
>>>> ### supervisor.* configs are for node supervisors
>>>
>>> # Define the amount of workers that can be run on this machine. Each
>>>> worker is assigned a port to use for communication
>>>
>>> supervisor.slots.ports:
>>>
>>> - 6700
>>>
>>> - 6701
>>>
>>> - 6702
>>>
>>> - 6703
>>>
>>> supervisor.childopts: "-Xmx256m"
>>>
>>> #how long supervisor will wait to ensure that a worker process is started
>>>
>>> supervisor.worker.start.timeout.secs: 120
>>>
>>> #how long between heartbeats until supervisor considers that worker dead
>>>> and tries to restart it
>>>
>>> supervisor.worker.timeout.secs: 30
>>>
>>> #how frequently the supervisor checks on the status of the processes
>>>> it's monitoring and restarts if necessary
>>>
>>> supervisor.monitor.frequency.secs: 3
>>>
>>> #how frequently the supervisor heartbeats to the cluster state (for
>>>> nimbus)
>>>
>>> supervisor.heartbeat.frequency.secs: 5
>>>
>>> supervisor.enable: true
>>>
>>>
>>>> ### worker.* configs are for task workers
>>>
>>> worker.childopts: "-Xmx768m"
>>>
>>> worker.heartbeat.frequency.secs: 1
>>>
>>>
>>>> task.heartbeat.frequency.secs: 3
>>>
>>> task.refresh.poll.secs: 10
>>>
>>>
>>>> zmq.threads: 1
>>>
>>> zmq.linger.millis: 5000
>>>
>>> zmq.hwm: 0
>>>
>>>
>>>>
>>>> storm.messaging.netty.server_worker_threads: 1
>>>
>>> storm.messaging.netty.client_worker_threads: 1
>>>
>>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>>
>>> storm.messaging.netty.max_retries: 30
>>>
>>> storm.messaging.netty.max_wait_ms: 1000
>>>
>>> storm.messaging.netty.min_wait_ms: 100
>>>
>>>
>>>> ### topology.* configs are for specific executing storms
>>>
>>> topology.enable.message.timeouts: true
>>>
>>> topology.debug: false
>>>
>>> topology.optimize: true
>>>
>>> topology.workers: 1
>>>
>>> topology.acker.executors: null
>>>
>>> topology.tasks: null
>>>
>>> # maximum amount of time a message has to complete before it's
>>>> considered failed
>>>
>>> topology.message.timeout.secs: 30
>>>
>>> topology.skip.missing.kryo.registrations: false
>>>
>>> topology.max.task.parallelism: null
>>>
>>> topology.max.spout.pending: null
>>>
>>> topology.state.synchronization.timeout.secs: 60
>>>
>>> topology.stats.sample.rate: 0.05
>>>
>>> topology.builtin.metrics.bucket.size.secs: 60
>>>
>>> topology.fall.back.on.java.serialization: true
>>>
>>> topology.worker.childopts: null
>>>
>>> topology.executor.receive.buffer.size: 1024 #batched
>>>
>>> topology.executor.send.buffer.size: 1024 #individual messages
>>>
>>> topology.receiver.buffer.size: 8 # setting it too high causes a lot of
>>>> problems (heartbeat thread gets starved, throughput plummets)
>>>
>>> topology.transfer.buffer.size: 1024 # batched
>>>
>>> topology.tick.tuple.freq.secs: null
>>>
>>> topology.worker.shared.thread.pool.size: 4
>>>
>>> topology.disruptor.wait.strategy:
>>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>>
>>> topology.spout.wait.strategy:
>>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>>
>>> topology.sleep.spout.wait.strategy.time.ms: 1
>>>
>>> topology.error.throttle.interval.secs: 10
>>>
>>> topology.max.error.report.per.interval: 5
>>>
>>> topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
>>>
>>> topology.tuple.serializer:
>>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>>
>>> topology.trident.batch.emit.interval.millis: 500
>>>
>>>
>>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>>
>>>
>>
>> bin/storm nimbus &
>>
>> bin/storm supervisor &
>>
>> bin/storm ui &
>>
>> *I launch another basic EC2 CentOS machine and do the exact same commands
>> except that I don't install zookeeper and I only run "bin/storm nimbus &"
>> at the end*
>>
>> Any thoughts would be greatly appreciated. Thanks so much for all of your
>> help. I'm sure someone else has done this before!
>>
>> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
>> wrote:
>>
>>> Although, implementing the Storm cluster manually would be really nice
>>> learning, I would suggest using HortonWorks distribution which comes with
>>> Storm as OOTB solution and you can configure everything from Ambari UI. We
>>> are using Storm on Amazon EC2 machine, though it is right now in beta
>>> stage. We are going to move to production in coming 2-3 months.
>>>
>>>
>>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>>> hartzell.stephen@gmail.com> wrote:
>>>
>>>> All,
>>>>
>>>> I implemented the suggestions given by Parh and Harsha. I am now
>>>> using the default.yaml but I changed the storm.zookeeper.servers to the
>>>> nimbus machine's ip address: 54.68.149.181. I also changed the nimbus.host
>>>> to 54.68.149.181. I also opened up port 6627. Now, the UI web page gives
>>>> the following error: org.apache.thrift7.transport.TTransportException:
>>>> java.net.ConnectException: Connection refused
>>>>
>>>> You should be able to see the error it gives by going to the web page
>>>> yourself at: http://54.68.149.181:8080. I am only using this account
>>>> to test and see if I can even get storm to work, so these machines are only
>>>> for testing. Perhaps someone could tell me what the storm.yaml file should
>>>> look like for this setup?
>>>>
>>>> -Thanks, Stephne
>>>>
>>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>>> hartzell.stephen@gmail.com> wrote:
>>>>
>>>>> I'm getting kind of confused by the storm.yaml file. Should I be using
>>>>> the default.yaml and just modify the zookeeper and nimbus ip, or should I
>>>>> use a bran new storm.yaml?
>>>>>
>>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is
>>>>> on the nimbus machine. what should the storm.yaml look like on my worker
>>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>>> very basic cluster up and running.
>>>>>
>>>>> -Thanks, Stephen
>>>>>
>>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>>> hartzell.stephen@gmail.com> wrote:
>>>>>
>>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>>> nothing got it working for me!
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>>
>>>>>>> The worker connects to the thrift port and not the ui port. You need
>>>>>>> to open port 6627 or whatever is the value being set in storm.yaml using
>>>>>>> property “nimbus.thrift.port”.
>>>>>>>
>>>>>>> Based on the configuration that you have pointed so far it seems
>>>>>>> your nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Parth
>>>>>>>
>>>>>>>
>>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>
>>>>>>> storm.zookeeper.servers:
>>>>>>> - "127.0.0.1"
>>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>>> machine is restarted)*
>>>>>>>
>>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip
>>>>>>> not 127.0.0.1.
>>>>>>>
>>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>>> 54.68.149.181"
>>>>>>> you are not able to reach this ip form worker node but able to
>>>>>>> access the UI using it?
>>>>>>> -Harsha
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>>
>>>>>>> Harsha,
>>>>>>>
>>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>>
>>>>>>> storm.zookeeper.servers:
>>>>>>> - "127.0.0.1"
>>>>>>>
>>>>>>>
>>>>>>> nimbus.host: "127.0.0.1"
>>>>>>>
>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>
>>>>>>>
>>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>>
>>>>>>> storm.zookeeper.servers:
>>>>>>> - "54.68.149.181"
>>>>>>>
>>>>>>>
>>>>>>> nimbus.host: "54.68.149.181"
>>>>>>>
>>>>>>> storm.local.dir: /tmp/storm
>>>>>>>
>>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>>> host.
>>>>>>> -Thanks, Stephen
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>
>>>>>>>
>>>>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>>>>> using the same storm.yaml for both the machines and also are you able to
>>>>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>>> -Harsha
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>>
>>>>>>> Harsha,
>>>>>>>
>>>>>>> Thanks so much for getting back with me. I will check the logs,
>>>>>>> but I don't seem to get any error messages. I have a nimbus AWS machine
>>>>>>> with zookeeper on it and a worker AWS machine.
>>>>>>>
>>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>>
>>>>>>> bin/storm nimbus &
>>>>>>> bin/storm supervisor &
>>>>>>> bin/storm ui
>>>>>>>
>>>>>>> On the worker machine I run:
>>>>>>> bin/storm supervisor
>>>>>>>
>>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>>>> the nimbus machine.
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Stephen,
>>>>>>> What are the issues you are seeing.
>>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in
>>>>>>> the storm configuration file"
>>>>>>>
>>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able
>>>>>>> to connect to your zookeeper cluster.
>>>>>>> Check your logs under storm_inst/logs/ for any errors when you try
>>>>>>> to start nimbus or supervisors.
>>>>>>> If you are installing it manually try following these steps if you
>>>>>>> are not already done.
>>>>>>>
>>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>>> -Harsha
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> I would greatly appreciate any help that anyone would afford. I've
>>>>>>> been trying to setup a storm cluster on AWS for a few weeks now on centOS
>>>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can get
>>>>>>> a supervisor and nimbus to run on a single machine, but I can't figure out
>>>>>>> how to get another worker to connect to nimbus. How do worker machines
>>>>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>>>>> I've gone through many tutorials and the official documentation, but this
>>>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>>>>> guy like me.
>>>>>>>
>>>>>>> Some of you may be tempted to point me toward storm-deploy, but I
>>>>>>> spent four days trying to get that to work until I gave up. I'm having
>>>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>>>> offer me any inputs you can!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>> entity to which it is addressed and may contain information that is
>>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>>> you have received this communication in error, please contact the sender
>>>>>>> immediately and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Vikas Agarwal
>>> 91 – 9928301411
>>>
>>> InfoObjects, Inc.
>>> Execution Matters
>>> http://www.infoobjects.com
>>> 2041 Mission College Boulevard, #280
>>> Santa Clara, CA 95054
>>> +1 (408) 988-2000 Work
>>> +1 (408) 716-2726 Fax
>>>
>>>
>>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: How Do Workers Connect To Nimbus
Posted by Vikas Agarwal <vi...@infoobjects.com>.
>>*I launch another basic EC2 CentOS machine and do the exact same commands
except that I don't install zookeeper and I only run "bin/storm nimbus &"
at the end*<<
Why would you run nimbus on two machines?
Yes, by default Hortonworks provides one supervisor node par machine, so I
am trying to add new EC2 machine with supervisor role. And it worked upto
some extent but still not fully functional. I am still testing its linear
scalability.
On Tue, Sep 9, 2014 at 5:15 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> Vikas,
>
>
> I've tried to use the HortonWorks distribution, but that only provides
> one supervisor and nimbus on one virtual machine. I'm excited to hear that
> you have storm working on AWS EC2 machines because that is exactly what I
> am trying to do! Right now we're still in the development stage, so all we
> are trying to do is to have one worker machine connect to one nimbus
> machine. So far we haven't got this work.
>
> Although it might be lengthy, let me go ahead and post the commands I'm
> using to setup the nimbus machine.
>
>
> *I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to TCP
> connections (it's public IP is 54.68.149.181)*
>
> sudo yum update
>
> sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
>
> *# Install zeromq*
>
> wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
>
> tar –zxvf zeromq-2.1.7.tar.gz
>
> cd zeromq-2.1.7.tar.gz
>
> ./configure
>
> make
>
> sudo make install
>
> cd ~
>
> *# Install jzmq*
>
> sudo yum install git java-devel libtool
>
> export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *# JDK
> required for jzmq in the configuration stage*
>
> cd $JAVA_HOME
>
> sudo ln –s include include *# JDK
> include directory has headers required for jzmq*
>
> cd ~
>
> git clone https://github.com/nathanmarz/jzmq.git
>
> cd jzmq/src/
>
> CLASSPATH=.:./.:$CLASSPATH
>
> touch classdist_noinst.stamp
>
> javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
> org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
> org/zeromq/ZMQStreamer.java
>
> cd ~/jzmq
>
> ./autogen.sh
>
> ./configure
>
> make
>
> sudo make install
>
> cd ~
>
> *# Download zookeeper*
>
> wget
> http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
>
> tar -zxvf zookeeper-3.4.6.tar.gz
>
> cd zookeeper-3.4.6
>
> vi conf/zoo.cfg
>
>
> tickTime=2000
>
> dataDir=/tmp/zookeeper
>
> clientPort=2181
>
>
> mkdir /tmp/zookeeper
> bin/zkServer.sh start
> bin/zkCli.sh -server 54.68.149.181:2181
>
>
> *# Download storm and modify configuration file (changes over default.yaml
> shown in bold)*
>
> https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
> storm-0.8.2.zip
> <https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzipstorm-0.8.2.zip>
>
> cd storm-0.8.2.zip
>
> vi conf/storm.yaml
>
>
> # Licensed to the Apache Software Foundation (ASF) under one
>>
>> # or more contributor license agreements. See the NOTICE file
>>
>> # distributed with this work for additional information
>>
>> # regarding copyright ownership. The ASF licenses this file
>>
>> # to you under the Apache License, Version 2.0 (the
>>
>> # "License"); you may not use this file except in compliance
>>
>> # with the License. You may obtain a copy of the License at
>>
>> #
>>
>> # http:# www.apache.org/licenses/LICENSE-2.0
>>
>> #
>>
>> # Unless required by applicable law or agreed to in writing, software
>>
>> # distributed under the License is distributed on an "AS IS" BASIS,
>>
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>>
>> # See the License for the specific language governing permissions and
>>
>> # limitations under the License.
>>
>>
>>>
>>> ########### These all have default values as shown
>>
>> ########### Additional configuration goes into storm.yaml
>>
>>
>>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>>> */home/ec2-user*"
>>
>>
>>> ### storm.* configs are general configurations
>>
>> # the local dir is where jars are kept
>>
>> storm.local.dir: "storm-local"
>>
>> storm.zookeeper.servers:
>>
>> - "*54.68.149.181*"
>>
>> storm.zookeeper.port: 2181
>>
>> storm.zookeeper.root: "/storm"
>>
>> storm.zookeeper.session.timeout: 20000
>>
>> storm.zookeeper.connection.timeout: 15000
>>
>> storm.zookeeper.retry.times: 5
>>
>> storm.zookeeper.retry.interval: 1000
>>
>> storm.zookeeper.retry.intervalceiling.millis: 30000
>>
>> storm.cluster.mode: "distributed" # can be distributed or local
>>
>> storm.local.mode.zmq: false
>>
>> storm.thrift.transport:
>>> "backtype.storm.security.auth.SimpleTransportPlugin"
>>
>> storm.messaging.transport: "backtype.storm.messaging.zmq"
>>
>>
>>> ### nimbus.* configs are for the master
>>
>> nimbus.host: "*54.68.149.181*"
>>
>> nimbus.thrift.port: 6627
>>
>> nimbus.childopts: "-Xmx1024m"
>>
>> nimbus.task.timeout.secs: 30
>>
>> nimbus.supervisor.timeout.secs: 60
>>
>> nimbus.monitor.freq.secs: 10
>>
>> nimbus.cleanup.inbox.freq.secs: 600
>>
>> nimbus.inbox.jar.expiration.secs: 3600
>>
>> nimbus.task.launch.secs: 120
>>
>> nimbus.reassign: true
>>
>> nimbus.file.copy.expiration.secs: 600
>>
>> nimbus.topology.validator:
>>> "backtype.storm.nimbus.DefaultTopologyValidator"
>>
>>
>>> ### ui.* configs are for the master
>>
>> ui.port: 8080
>>
>> ui.childopts: "-Xmx768m"
>>
>>
>>> logviewer.port: 8000
>>
>> logviewer.childopts: "-Xmx128m"
>>
>> logviewer.appender.name: "A1"
>>
>>
>>>
>>> drpc.port: 3772
>>
>> drpc.worker.threads: 64
>>
>> drpc.queue.size: 128
>>
>> drpc.invocations.port: 3773
>>
>> drpc.request.timeout.secs: 600
>>
>> drpc.childopts: "-Xmx768m"
>>
>>
>>> transactional.zookeeper.root: "/transactional"
>>
>> transactional.zookeeper.servers: null
>>
>> transactional.zookeeper.port: null
>>
>>
>>> ### supervisor.* configs are for node supervisors
>>
>> # Define the amount of workers that can be run on this machine. Each
>>> worker is assigned a port to use for communication
>>
>> supervisor.slots.ports:
>>
>> - 6700
>>
>> - 6701
>>
>> - 6702
>>
>> - 6703
>>
>> supervisor.childopts: "-Xmx256m"
>>
>> #how long supervisor will wait to ensure that a worker process is started
>>
>> supervisor.worker.start.timeout.secs: 120
>>
>> #how long between heartbeats until supervisor considers that worker dead
>>> and tries to restart it
>>
>> supervisor.worker.timeout.secs: 30
>>
>> #how frequently the supervisor checks on the status of the processes it's
>>> monitoring and restarts if necessary
>>
>> supervisor.monitor.frequency.secs: 3
>>
>> #how frequently the supervisor heartbeats to the cluster state (for
>>> nimbus)
>>
>> supervisor.heartbeat.frequency.secs: 5
>>
>> supervisor.enable: true
>>
>>
>>> ### worker.* configs are for task workers
>>
>> worker.childopts: "-Xmx768m"
>>
>> worker.heartbeat.frequency.secs: 1
>>
>>
>>> task.heartbeat.frequency.secs: 3
>>
>> task.refresh.poll.secs: 10
>>
>>
>>> zmq.threads: 1
>>
>> zmq.linger.millis: 5000
>>
>> zmq.hwm: 0
>>
>>
>>>
>>> storm.messaging.netty.server_worker_threads: 1
>>
>> storm.messaging.netty.client_worker_threads: 1
>>
>> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>>
>> storm.messaging.netty.max_retries: 30
>>
>> storm.messaging.netty.max_wait_ms: 1000
>>
>> storm.messaging.netty.min_wait_ms: 100
>>
>>
>>> ### topology.* configs are for specific executing storms
>>
>> topology.enable.message.timeouts: true
>>
>> topology.debug: false
>>
>> topology.optimize: true
>>
>> topology.workers: 1
>>
>> topology.acker.executors: null
>>
>> topology.tasks: null
>>
>> # maximum amount of time a message has to complete before it's considered
>>> failed
>>
>> topology.message.timeout.secs: 30
>>
>> topology.skip.missing.kryo.registrations: false
>>
>> topology.max.task.parallelism: null
>>
>> topology.max.spout.pending: null
>>
>> topology.state.synchronization.timeout.secs: 60
>>
>> topology.stats.sample.rate: 0.05
>>
>> topology.builtin.metrics.bucket.size.secs: 60
>>
>> topology.fall.back.on.java.serialization: true
>>
>> topology.worker.childopts: null
>>
>> topology.executor.receive.buffer.size: 1024 #batched
>>
>> topology.executor.send.buffer.size: 1024 #individual messages
>>
>> topology.receiver.buffer.size: 8 # setting it too high causes a lot of
>>> problems (heartbeat thread gets starved, throughput plummets)
>>
>> topology.transfer.buffer.size: 1024 # batched
>>
>> topology.tick.tuple.freq.secs: null
>>
>> topology.worker.shared.thread.pool.size: 4
>>
>> topology.disruptor.wait.strategy:
>>> "com.lmax.disruptor.BlockingWaitStrategy"
>>
>> topology.spout.wait.strategy:
>>> "backtype.storm.spout.SleepSpoutWaitStrategy"
>>
>> topology.sleep.spout.wait.strategy.time.ms: 1
>>
>> topology.error.throttle.interval.secs: 10
>>
>> topology.max.error.report.per.interval: 5
>>
>> topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
>>
>> topology.tuple.serializer:
>>> "backtype.storm.serialization.types.ListDelegateSerializer"
>>
>> topology.trident.batch.emit.interval.millis: 500
>>
>>
>>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>>
>>
>
> bin/storm nimbus &
>
> bin/storm supervisor &
>
> bin/storm ui &
>
> *I launch another basic EC2 CentOS machine and do the exact same commands
> except that I don't install zookeeper and I only run "bin/storm nimbus &"
> at the end*
>
> Any thoughts would be greatly appreciated. Thanks so much for all of your
> help. I'm sure someone else has done this before!
>
> On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
> wrote:
>
>> Although, implementing the Storm cluster manually would be really nice
>> learning, I would suggest using HortonWorks distribution which comes with
>> Storm as OOTB solution and you can configure everything from Ambari UI. We
>> are using Storm on Amazon EC2 machine, though it is right now in beta
>> stage. We are going to move to production in coming 2-3 months.
>>
>>
>> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> All,
>>>
>>> I implemented the suggestions given by Parh and Harsha. I am now using
>>> the default.yaml but I changed the storm.zookeeper.servers to the nimbus
>>> machine's ip address: 54.68.149.181. I also changed the nimbus.host to
>>> 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the
>>> following error: org.apache.thrift7.transport.TTransportException:
>>> java.net.ConnectException: Connection refused
>>>
>>> You should be able to see the error it gives by going to the web page
>>> yourself at: http://54.68.149.181:8080. I am only using this account to
>>> test and see if I can even get storm to work, so these machines are only
>>> for testing. Perhaps someone could tell me what the storm.yaml file should
>>> look like for this setup?
>>>
>>> -Thanks, Stephne
>>>
>>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>>> hartzell.stephen@gmail.com> wrote:
>>>
>>>> I'm getting kind of confused by the storm.yaml file. Should I be using
>>>> the default.yaml and just modify the zookeeper and nimbus ip, or should I
>>>> use a bran new storm.yaml?
>>>>
>>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is
>>>> on the nimbus machine. what should the storm.yaml look like on my worker
>>>> and nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>>> very basic cluster up and running.
>>>>
>>>> -Thanks, Stephen
>>>>
>>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>>> hartzell.stephen@gmail.com> wrote:
>>>>
>>>>> All Thanks so much for your help. I cannot tell you how much I
>>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>>> get this to work. I'll let you know what happens after I try to implement
>>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>>> common use-case. I've read and watched everything I can on the topic and
>>>>> nothing got it working for me!
>>>>>
>>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>>
>>>>>> The worker connects to the thrift port and not the ui port. You need
>>>>>> to open port 6627 or whatever is the value being set in storm.yaml using
>>>>>> property “nimbus.thrift.port”.
>>>>>>
>>>>>> Based on the configuration that you have pointed so far it seems your
>>>>>> nimbus host has nimbus,ui,supervisor working because you actually have
>>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>>
>>>>>> Thanks
>>>>>> Parth
>>>>>>
>>>>>>
>>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>>
>>>>>> storm.zookeeper.servers:
>>>>>> - "127.0.0.1"
>>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>>> machine is restarted)*
>>>>>>
>>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>>>>>> 127.0.0.1.
>>>>>>
>>>>>> "No, I cannot ping my host which has a public ip address of
>>>>>> 54.68.149.181"
>>>>>> you are not able to reach this ip form worker node but able to access
>>>>>> the UI using it?
>>>>>> -Harsha
>>>>>>
>>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>>
>>>>>> Harsha,
>>>>>>
>>>>>> The storm.yaml on the host machine looks like this:
>>>>>>
>>>>>> storm.zookeeper.servers:
>>>>>> - "127.0.0.1"
>>>>>>
>>>>>>
>>>>>> nimbus.host: "127.0.0.1"
>>>>>>
>>>>>> storm.local.dir: /tmp/storm
>>>>>>
>>>>>>
>>>>>> The storm.yaml on the worker machine looks like this:
>>>>>>
>>>>>> storm.zookeeper.servers:
>>>>>> - "54.68.149.181"
>>>>>>
>>>>>>
>>>>>> nimbus.host: "54.68.149.181"
>>>>>>
>>>>>> storm.local.dir: /tmp/storm
>>>>>>
>>>>>> No, I cannot ping my host which has a public ip address of
>>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>>> host.
>>>>>> -Thanks, Stephen
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>>
>>>>>>
>>>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>>>> using the same storm.yaml for both the machines and also are you able to
>>>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>>>> -Harsha
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>>
>>>>>> Harsha,
>>>>>>
>>>>>> Thanks so much for getting back with me. I will check the logs, but
>>>>>> I don't seem to get any error messages. I have a nimbus AWS machine with
>>>>>> zookeeper on it and a worker AWS machine.
>>>>>>
>>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>>
>>>>>> bin/storm nimbus &
>>>>>> bin/storm supervisor &
>>>>>> bin/storm ui
>>>>>>
>>>>>> On the worker machine I run:
>>>>>> bin/storm supervisor
>>>>>>
>>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>>> the nimbus machine.
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>>
>>>>>>
>>>>>> Hi Stephen,
>>>>>> What are the issues you are seeing.
>>>>>> "How do worker machines "know" how to connect to nimbus? Is it in the
>>>>>> storm configuration file"
>>>>>>
>>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>>>>>> connect to your zookeeper cluster.
>>>>>> Check your logs under storm_inst/logs/ for any errors when you try to
>>>>>> start nimbus or supervisors.
>>>>>> If you are installing it manually try following these steps if you
>>>>>> are not already done.
>>>>>>
>>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>>> -Harsha
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>>
>>>>>> All,
>>>>>>
>>>>>> I would greatly appreciate any help that anyone would afford. I've
>>>>>> been trying to setup a storm cluster on AWS for a few weeks now on centOS
>>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can get
>>>>>> a supervisor and nimbus to run on a single machine, but I can't figure out
>>>>>> how to get another worker to connect to nimbus. How do worker machines
>>>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>>>> I've gone through many tutorials and the official documentation, but this
>>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>>>> guy like me.
>>>>>>
>>>>>> Some of you may be tempted to point me toward storm-deploy, but I
>>>>>> spent four days trying to get that to work until I gave up. I'm having
>>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>>> offer me any inputs you can!
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to which it is addressed and may contain information that is
>>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>>> notified that any printing, copying, dissemination, distribution,
>>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>>> you have received this communication in error, please contact the sender
>>>>>> immediately and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>> Vikas Agarwal
>> 91 – 9928301411
>>
>> InfoObjects, Inc.
>> Execution Matters
>> http://www.infoobjects.com
>> 2041 Mission College Boulevard, #280
>> Santa Clara, CA 95054
>> +1 (408) 988-2000 Work
>> +1 (408) 716-2726 Fax
>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
Vikas,
I've tried to use the HortonWorks distribution, but that only provides
one supervisor and nimbus on one virtual machine. I'm excited to hear that
you have storm working on AWS EC2 machines because that is exactly what I
am trying to do! Right now we're still in the development stage, so all we
are trying to do is to have one worker machine connect to one nimbus
machine. So far we haven't got this work.
Although it might be lengthy, let me go ahead and post the commands I'm
using to setup the nimbus machine.
*I launch a basic EC2 CentOS machine with ports 6627 and 8080 open to TCP
connections (it's public IP is 54.68.149.181)*
sudo yum update
sudo yum install libuuid-devel gcc gcc-c++ kernel-devel
*# Install zeromq*
wget http://download.zeromq.org/zeromq-2.1.7.tar.gz
tar –zxvf zeromq-2.1.7.tar.gz
cd zeromq-2.1.7.tar.gz
./configure
make
sudo make install
cd ~
*# Install jzmq*
sudo yum install git java-devel libtool
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.x.x86_64 *# JDK
required for jzmq in the configuration stage*
cd $JAVA_HOME
sudo ln –s include include *# JDK
include directory has headers required for jzmq*
cd ~
git clone https://github.com/nathanmarz/jzmq.git
cd jzmq/src/
CLASSPATH=.:./.:$CLASSPATH
touch classdist_noinst.stamp
javac -d . org/zeromq/ZMQ.java org/zeromq/ZMQException.java
org/zeromq/ZMQQueue.java org/zeromq/ZMQForwarder.java
org/zeromq/ZMQStreamer.java
cd ~/jzmq
./autogen.sh
./configure
make
sudo make install
cd ~
*# Download zookeeper*
wget
http://mirror.cc.columbia.edu/pub/software/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
tar -zxvf zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6
vi conf/zoo.cfg
tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181
mkdir /tmp/zookeeper
bin/zkServer.sh start
bin/zkCli.sh -server 54.68.149.181:2181
*# Download storm and modify configuration file (changes over default.yaml
shown in bold)*
https://www.dropbox.com/s/fl4kr7w0oc8ihdw/storm-0.8.2.zipunzip
storm-0.8.2.zip
cd storm-0.8.2.zip
vi conf/storm.yaml
# Licensed to the Apache Software Foundation (ASF) under one
>
> # or more contributor license agreements. See the NOTICE file
>
> # distributed with this work for additional information
>
> # regarding copyright ownership. The ASF licenses this file
>
> # to you under the Apache License, Version 2.0 (the
>
> # "License"); you may not use this file except in compliance
>
> # with the License. You may obtain a copy of the License at
>
> #
>
> # http:# www.apache.org/licenses/LICENSE-2.0
>
> #
>
> # Unless required by applicable law or agreed to in writing, software
>
> # distributed under the License is distributed on an "AS IS" BASIS,
>
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>
> # See the License for the specific language governing permissions and
>
> # limitations under the License.
>
>
>>
>> ########### These all have default values as shown
>
> ########### Additional configuration goes into storm.yaml
>
>
>> java.library.path: "/usr/local/lib:/opt/local/lib:/usr/lib:
>> */home/ec2-user*"
>
>
>> ### storm.* configs are general configurations
>
> # the local dir is where jars are kept
>
> storm.local.dir: "storm-local"
>
> storm.zookeeper.servers:
>
> - "*54.68.149.181*"
>
> storm.zookeeper.port: 2181
>
> storm.zookeeper.root: "/storm"
>
> storm.zookeeper.session.timeout: 20000
>
> storm.zookeeper.connection.timeout: 15000
>
> storm.zookeeper.retry.times: 5
>
> storm.zookeeper.retry.interval: 1000
>
> storm.zookeeper.retry.intervalceiling.millis: 30000
>
> storm.cluster.mode: "distributed" # can be distributed or local
>
> storm.local.mode.zmq: false
>
> storm.thrift.transport:
>> "backtype.storm.security.auth.SimpleTransportPlugin"
>
> storm.messaging.transport: "backtype.storm.messaging.zmq"
>
>
>> ### nimbus.* configs are for the master
>
> nimbus.host: "*54.68.149.181*"
>
> nimbus.thrift.port: 6627
>
> nimbus.childopts: "-Xmx1024m"
>
> nimbus.task.timeout.secs: 30
>
> nimbus.supervisor.timeout.secs: 60
>
> nimbus.monitor.freq.secs: 10
>
> nimbus.cleanup.inbox.freq.secs: 600
>
> nimbus.inbox.jar.expiration.secs: 3600
>
> nimbus.task.launch.secs: 120
>
> nimbus.reassign: true
>
> nimbus.file.copy.expiration.secs: 600
>
> nimbus.topology.validator: "backtype.storm.nimbus.DefaultTopologyValidator"
>
>
>> ### ui.* configs are for the master
>
> ui.port: 8080
>
> ui.childopts: "-Xmx768m"
>
>
>> logviewer.port: 8000
>
> logviewer.childopts: "-Xmx128m"
>
> logviewer.appender.name: "A1"
>
>
>>
>> drpc.port: 3772
>
> drpc.worker.threads: 64
>
> drpc.queue.size: 128
>
> drpc.invocations.port: 3773
>
> drpc.request.timeout.secs: 600
>
> drpc.childopts: "-Xmx768m"
>
>
>> transactional.zookeeper.root: "/transactional"
>
> transactional.zookeeper.servers: null
>
> transactional.zookeeper.port: null
>
>
>> ### supervisor.* configs are for node supervisors
>
> # Define the amount of workers that can be run on this machine. Each
>> worker is assigned a port to use for communication
>
> supervisor.slots.ports:
>
> - 6700
>
> - 6701
>
> - 6702
>
> - 6703
>
> supervisor.childopts: "-Xmx256m"
>
> #how long supervisor will wait to ensure that a worker process is started
>
> supervisor.worker.start.timeout.secs: 120
>
> #how long between heartbeats until supervisor considers that worker dead
>> and tries to restart it
>
> supervisor.worker.timeout.secs: 30
>
> #how frequently the supervisor checks on the status of the processes it's
>> monitoring and restarts if necessary
>
> supervisor.monitor.frequency.secs: 3
>
> #how frequently the supervisor heartbeats to the cluster state (for nimbus)
>
> supervisor.heartbeat.frequency.secs: 5
>
> supervisor.enable: true
>
>
>> ### worker.* configs are for task workers
>
> worker.childopts: "-Xmx768m"
>
> worker.heartbeat.frequency.secs: 1
>
>
>> task.heartbeat.frequency.secs: 3
>
> task.refresh.poll.secs: 10
>
>
>> zmq.threads: 1
>
> zmq.linger.millis: 5000
>
> zmq.hwm: 0
>
>
>>
>> storm.messaging.netty.server_worker_threads: 1
>
> storm.messaging.netty.client_worker_threads: 1
>
> storm.messaging.netty.buffer_size: 5242880 #5MB buffer
>
> storm.messaging.netty.max_retries: 30
>
> storm.messaging.netty.max_wait_ms: 1000
>
> storm.messaging.netty.min_wait_ms: 100
>
>
>> ### topology.* configs are for specific executing storms
>
> topology.enable.message.timeouts: true
>
> topology.debug: false
>
> topology.optimize: true
>
> topology.workers: 1
>
> topology.acker.executors: null
>
> topology.tasks: null
>
> # maximum amount of time a message has to complete before it's considered
>> failed
>
> topology.message.timeout.secs: 30
>
> topology.skip.missing.kryo.registrations: false
>
> topology.max.task.parallelism: null
>
> topology.max.spout.pending: null
>
> topology.state.synchronization.timeout.secs: 60
>
> topology.stats.sample.rate: 0.05
>
> topology.builtin.metrics.bucket.size.secs: 60
>
> topology.fall.back.on.java.serialization: true
>
> topology.worker.childopts: null
>
> topology.executor.receive.buffer.size: 1024 #batched
>
> topology.executor.send.buffer.size: 1024 #individual messages
>
> topology.receiver.buffer.size: 8 # setting it too high causes a lot of
>> problems (heartbeat thread gets starved, throughput plummets)
>
> topology.transfer.buffer.size: 1024 # batched
>
> topology.tick.tuple.freq.secs: null
>
> topology.worker.shared.thread.pool.size: 4
>
> topology.disruptor.wait.strategy: "com.lmax.disruptor.BlockingWaitStrategy"
>
> topology.spout.wait.strategy: "backtype.storm.spout.SleepSpoutWaitStrategy"
>
> topology.sleep.spout.wait.strategy.time.ms: 1
>
> topology.error.throttle.interval.secs: 10
>
> topology.max.error.report.per.interval: 5
>
> topology.kryo.factory: "backtype.storm.serialization.DefaultKryoFactory"
>
> topology.tuple.serializer:
>> "backtype.storm.serialization.types.ListDelegateSerializer"
>
> topology.trident.batch.emit.interval.millis: 500
>
>
>> dev.zookeeper.path: "/tmp/dev-storm-zookeeper"
>
>
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui &
*I launch another basic EC2 CentOS machine and do the exact same commands
except that I don't install zookeeper and I only run "bin/storm nimbus &"
at the end*
Any thoughts would be greatly appreciated. Thanks so much for all of your
help. I'm sure someone else has done this before!
On Mon, Sep 8, 2014 at 11:14 PM, Vikas Agarwal <vi...@infoobjects.com>
wrote:
> Although, implementing the Storm cluster manually would be really nice
> learning, I would suggest using HortonWorks distribution which comes with
> Storm as OOTB solution and you can configure everything from Ambari UI. We
> are using Storm on Amazon EC2 machine, though it is right now in beta
> stage. We are going to move to production in coming 2-3 months.
>
>
> On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> All,
>>
>> I implemented the suggestions given by Parh and Harsha. I am now using
>> the default.yaml but I changed the storm.zookeeper.servers to the nimbus
>> machine's ip address: 54.68.149.181. I also changed the nimbus.host to
>> 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the
>> following error: org.apache.thrift7.transport.TTransportException:
>> java.net.ConnectException: Connection refused
>>
>> You should be able to see the error it gives by going to the web page
>> yourself at: http://54.68.149.181:8080. I am only using this account to
>> test and see if I can even get storm to work, so these machines are only
>> for testing. Perhaps someone could tell me what the storm.yaml file should
>> look like for this setup?
>>
>> -Thanks, Stephne
>>
>> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> I'm getting kind of confused by the storm.yaml file. Should I be using
>>> the default.yaml and just modify the zookeeper and nimbus ip, or should I
>>> use a bran new storm.yaml?
>>>
>>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on
>>> the nimbus machine. what should the storm.yaml look like on my worker and
>>> nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>>> machine? I am not trying to do anything fancy, I am just trying to get a
>>> very basic cluster up and running.
>>>
>>> -Thanks, Stephen
>>>
>>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>>> hartzell.stephen@gmail.com> wrote:
>>>
>>>> All Thanks so much for your help. I cannot tell you how much I
>>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>>> head again the wall : D. I've spent an enormous amount of time trying to
>>>> get this to work. I'll let you know what happens after I try to implement
>>>> your suggestions. It would be really cool if someone had a tutorial that
>>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>>> common use-case. I've read and watched everything I can on the topic and
>>>> nothing got it working for me!
>>>>
>>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>>> pbrahmbhatt@hortonworks.com> wrote:
>>>>
>>>>> The worker connects to the thrift port and not the ui port. You need
>>>>> to open port 6627 or whatever is the value being set in storm.yaml using
>>>>> property “nimbus.thrift.port”.
>>>>>
>>>>> Based on the configuration that you have pointed so far it seems your
>>>>> nimbus host has nimbus,ui,supervisor working because you actually have
>>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>>> change it to a value that is the public ip instead of loopback interface.
>>>>>
>>>>> Thanks
>>>>> Parth
>>>>>
>>>>>
>>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>>
>>>>> storm.zookeeper.servers:
>>>>> - "127.0.0.1"
>>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a
>>>>> different folder probably /home/storm, /tmp/storm will get deleted if your
>>>>> machine is restarted)*
>>>>>
>>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>>>>> 127.0.0.1.
>>>>>
>>>>> "No, I cannot ping my host which has a public ip address of
>>>>> 54.68.149.181"
>>>>> you are not able to reach this ip form worker node but able to access
>>>>> the UI using it?
>>>>> -Harsha
>>>>>
>>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>>
>>>>> Harsha,
>>>>>
>>>>> The storm.yaml on the host machine looks like this:
>>>>>
>>>>> storm.zookeeper.servers:
>>>>> - "127.0.0.1"
>>>>>
>>>>>
>>>>> nimbus.host: "127.0.0.1"
>>>>>
>>>>> storm.local.dir: /tmp/storm
>>>>>
>>>>>
>>>>> The storm.yaml on the worker machine looks like this:
>>>>>
>>>>> storm.zookeeper.servers:
>>>>> - "54.68.149.181"
>>>>>
>>>>>
>>>>> nimbus.host: "54.68.149.181"
>>>>>
>>>>> storm.local.dir: /tmp/storm
>>>>>
>>>>> No, I cannot ping my host which has a public ip address of
>>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>>> host.
>>>>> -Thanks, Stephen
>>>>>
>>>>>
>>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>>
>>>>>
>>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>>> using the same storm.yaml for both the machines and also are you able to
>>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>>> -Harsha
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>>
>>>>> Harsha,
>>>>>
>>>>> Thanks so much for getting back with me. I will check the logs, but
>>>>> I don't seem to get any error messages. I have a nimbus AWS machine with
>>>>> zookeeper on it and a worker AWS machine.
>>>>>
>>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>>
>>>>> bin/storm nimbus &
>>>>> bin/storm supervisor &
>>>>> bin/storm ui
>>>>>
>>>>> On the worker machine I run:
>>>>> bin/storm supervisor
>>>>>
>>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>>> the nimbus machine.
>>>>>
>>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>>
>>>>>
>>>>> Hi Stephen,
>>>>> What are the issues you are seeing.
>>>>> "How do worker machines "know" how to connect to nimbus? Is it in the
>>>>> storm configuration file"
>>>>>
>>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>>>>> connect to your zookeeper cluster.
>>>>> Check your logs under storm_inst/logs/ for any errors when you try to
>>>>> start nimbus or supervisors.
>>>>> If you are installing it manually try following these steps if you are
>>>>> not already done.
>>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>>> -Harsha
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>>
>>>>> All,
>>>>>
>>>>> I would greatly appreciate any help that anyone would afford. I've
>>>>> been trying to setup a storm cluster on AWS for a few weeks now on centOS
>>>>> EC2 machines. So far, I haven't been able to get a cluster built. I can get
>>>>> a supervisor and nimbus to run on a single machine, but I can't figure out
>>>>> how to get another worker to connect to nimbus. How do worker machines
>>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>>> I've gone through many tutorials and the official documentation, but this
>>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>>> guy like me.
>>>>>
>>>>> Some of you may be tempted to point me toward storm-deploy, but I
>>>>> spent four days trying to get that to work until I gave up. I'm having
>>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>>> offer me any inputs you can!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
Re: How Do Workers Connect To Nimbus
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Although, implementing the Storm cluster manually would be really nice
learning, I would suggest using HortonWorks distribution which comes with
Storm as OOTB solution and you can configure everything from Ambari UI. We
are using Storm on Amazon EC2 machine, though it is right now in beta
stage. We are going to move to production in coming 2-3 months.
On Tue, Sep 9, 2014 at 5:53 AM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> All,
>
> I implemented the suggestions given by Parh and Harsha. I am now using
> the default.yaml but I changed the storm.zookeeper.servers to the nimbus
> machine's ip address: 54.68.149.181. I also changed the nimbus.host to
> 54.68.149.181. I also opened up port 6627. Now, the UI web page gives the
> following error: org.apache.thrift7.transport.TTransportException:
> java.net.ConnectException: Connection refused
>
> You should be able to see the error it gives by going to the web page
> yourself at: http://54.68.149.181:8080. I am only using this account to
> test and see if I can even get storm to work, so these machines are only
> for testing. Perhaps someone could tell me what the storm.yaml file should
> look like for this setup?
>
> -Thanks, Stephne
>
> On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> I'm getting kind of confused by the storm.yaml file. Should I be using
>> the default.yaml and just modify the zookeeper and nimbus ip, or should I
>> use a bran new storm.yaml?
>>
>> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on
>> the nimbus machine. what should the storm.yaml look like on my worker and
>> nimbus machine? Will the storm.yaml be the same on my worker and nimbus
>> machine? I am not trying to do anything fancy, I am just trying to get a
>> very basic cluster up and running.
>>
>> -Thanks, Stephen
>>
>> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
>> hartzell.stephen@gmail.com> wrote:
>>
>>> All Thanks so much for your help. I cannot tell you how much I
>>> appreciate it. I'm going to try out your suggestions and keep banging my
>>> head again the wall : D. I've spent an enormous amount of time trying to
>>> get this to work. I'll let you know what happens after I try to implement
>>> your suggestions. It would be really cool if someone had a tutorial that
>>> detailed this part. (I'll make it myself if I ever get this to work!) It
>>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>>> common use-case. I've read and watched everything I can on the topic and
>>> nothing got it working for me!
>>>
>>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> The worker connects to the thrift port and not the ui port. You need to
>>>> open port 6627 or whatever is the value being set in storm.yaml using
>>>> property “nimbus.thrift.port”.
>>>>
>>>> Based on the configuration that you have pointed so far it seems your
>>>> nimbus host has nimbus,ui,supervisor working because you actually have
>>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>>> change it to a value that is the public ip instead of loopback interface.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>>
>>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>>
>>>> storm.zookeeper.servers:
>>>> - "127.0.0.1"
>>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>>> interface , instead either use your public ip or 0.0.0.0*)
>>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
>>>> folder probably /home/storm, /tmp/storm will get deleted if your machine is
>>>> restarted)*
>>>>
>>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>>>> 127.0.0.1.
>>>>
>>>> "No, I cannot ping my host which has a public ip address of
>>>> 54.68.149.181"
>>>> you are not able to reach this ip form worker node but able to access
>>>> the UI using it?
>>>> -Harsha
>>>>
>>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>>
>>>> Harsha,
>>>>
>>>> The storm.yaml on the host machine looks like this:
>>>>
>>>> storm.zookeeper.servers:
>>>> - "127.0.0.1"
>>>>
>>>>
>>>> nimbus.host: "127.0.0.1"
>>>>
>>>> storm.local.dir: /tmp/storm
>>>>
>>>>
>>>> The storm.yaml on the worker machine looks like this:
>>>>
>>>> storm.zookeeper.servers:
>>>> - "54.68.149.181"
>>>>
>>>>
>>>> nimbus.host: "54.68.149.181"
>>>>
>>>> storm.local.dir: /tmp/storm
>>>>
>>>> No, I cannot ping my host which has a public ip address of
>>>> 54.68.149.181 although I can connect to the UI web page when it is hosted.
>>>> I don't know how I would go about connecting to zookeeper on the nimbus
>>>> host.
>>>> -Thanks, Stephen
>>>>
>>>>
>>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>>
>>>>
>>>> There aren't any errors in worker machine supervisor logs. Are you
>>>> using the same storm.yaml for both the machines and also are you able to
>>>> ping your nimbus host or connect to zookeeper on nimbus host.
>>>> -Harsha
>>>>
>>>>
>>>>
>>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>>
>>>> Harsha,
>>>>
>>>> Thanks so much for getting back with me. I will check the logs, but I
>>>> don't seem to get any error messages. I have a nimbus AWS machine with
>>>> zookeeper on it and a worker AWS machine.
>>>>
>>>> On the nimbus machine I start the zookeeper and then I run:
>>>>
>>>> bin/storm nimbus &
>>>> bin/storm supervisor &
>>>> bin/storm ui
>>>>
>>>> On the worker machine I run:
>>>> bin/storm supervisor
>>>>
>>>> When I go to the UI page, I only see 1 supervisor (the one on the
>>>> nimbus machine). So apparently, the worker machine isn't "registering" with
>>>> the nimbus machine.
>>>>
>>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>>
>>>>
>>>> Hi Stephen,
>>>> What are the issues you are seeing.
>>>> "How do worker machines "know" how to connect to nimbus? Is it in the
>>>> storm configuration file"
>>>>
>>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>>>> connect to your zookeeper cluster.
>>>> Check your logs under storm_inst/logs/ for any errors when you try to
>>>> start nimbus or supervisors.
>>>> If you are installing it manually try following these steps if you are
>>>> not already done.
>>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>>> -Harsha
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>>
>>>> All,
>>>>
>>>> I would greatly appreciate any help that anyone would afford. I've been
>>>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
>>>> machines. So far, I haven't been able to get a cluster built. I can get a
>>>> supervisor and nimbus to run on a single machine, but I can't figure out
>>>> how to get another worker to connect to nimbus. How do worker machines
>>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>>> I've gone through many tutorials and the official documentation, but this
>>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>>> guy like me.
>>>>
>>>> Some of you may be tempted to point me toward storm-deploy, but I
>>>> spent four days trying to get that to work until I gave up. I'm having
>>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>>> offer me any inputs you can!
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to which it is addressed and may contain information that is
>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>> If the reader of this message is not the intended recipient, you are hereby
>>>> notified that any printing, copying, dissemination, distribution,
>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>> you have received this communication in error, please contact the sender
>>>> immediately and delete it from your system. Thank You.
>>>>
>>>
>>>
>>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: How Do Workers Connect To Nimbus
Posted by Harsha <st...@harsha.io>.
Stephen,
I am not able to reach that IP. But you shouldn't
modify the default.yaml just change storm.yaml under conf
"Will the storm.yaml be the same on my worker and nimbus
machine?"
it should be the same on both machines. Make sure your
zookeeper also running on that ip. And check for logs under you
storm installation it should be under logs dir.
-Harsha
On Mon, Sep 8, 2014, at 05:23 PM, Stephen Hartzell wrote:
All,
I implemented the suggestions given by Parh and Harsha. I am
now using the default.yaml but I changed the
storm.zookeeper.servers to the nimbus machine's ip address:
54.68.149.181. I also changed the nimbus.host to 54.68.149.181.
I also opened up port 6627. Now, the UI web page gives the
following
error: org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused
You should be able to see the error it gives by going to the
web page yourself at: [1]http://54.68.149.181:8080. I am only
using this account to test and see if I can even get storm to
work, so these machines are only for testing. Perhaps someone
could tell me what the storm.yaml file should look like for
this setup?
-Thanks, Stephne
On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell
<[2...@gmail.com> wrote:
I'm getting kind of confused by the storm.yaml file. Should I
be using the default.yaml and just modify the zookeeper and
nimbus ip, or should I use a bran new storm.yaml?
My nimbus machine has the ip address: 54.68.149.181.My
zookeeper is on the nimbus machine. what should the storm.yaml
look like on my worker and nimbus machine? Will the storm.yaml
be the same on my worker and nimbus machine? I am not trying to
do anything fancy, I am just trying to get a very basic cluster
up and running.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell
<[3...@gmail.com> wrote:
All Thanks so much for your help. I cannot tell you how much
I appreciate it. I'm going to try out your suggestions and keep
banging my head again the wall : D. I've spent an enormous
amount of time trying to get this to work. I'll let you know
what happens after I try to implement your suggestions. It
would be really cool if someone had a tutorial that detailed
this part. (I'll make it myself if I ever get this to work!) It
seems like trying to get a two-machine cluster setup on AWS
would be a VERY common use-case. I've read and watched
everything I can on the topic and nothing got it working for
me!
On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt
<[4...@hortonworks.com> wrote:
The worker connects to the thrift port and not the ui port. You
need to open port 6627 or whatever is the value being set in
storm.yaml using property “nimbus.thrift.port”.
Based on the configuration that you have pointed so far it
seems your nimbus host has nimbus,ui,supervisor working because
you actually have zookeeper running locally on that host. As
Harsha pointed out you need to change it to a value that is the
public ip instead of loopback interface.
Thanks
Parth
On Sep 8, 2014, at 3:42 PM, Harsha <[5...@harsha.io> wrote:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1" ( 127.0.0.1 causes to bind a loopback
interface , instead either use your public ip or 0.0.0.0)
storm.local.dir: /tmp/storm ( I recommend this to move to a
different folder probably /home/storm, /tmp/storm will get
deleted if your machine is restarted)
make sure you zookeeper is also listening in 0.0.0.0 or public
ip not 127.0.0.1.
"No, I cannot ping my host which has a public ip address of
54.68.149.181"
you are not able to reach this ip form worker node but able to
access the UI using it?
-Harsha
On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
Harsha,
The storm.yaml on the host machine looks like this:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1"
storm.local.dir: /tmp/storm
The storm.yaml on the worker machine looks like this:
storm.zookeeper.servers:
- "54.68.149.181"
nimbus.host: "54.68.149.181"
storm.local.dir: /tmp/storm
No, I cannot ping my host which has a public ip address of
54.68.149.181 although I can connect to the UI web page when it
is hosted. I don't know how I would go about connecting to
zookeeper on the nimbus host.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[6...@harsha.io>
wrote:
There aren't any errors in worker machine supervisor logs. Are
you using the same storm.yaml for both the machines and also
are you able to ping your nimbus host or connect to zookeeper
on nimbus host.
-Harsha
On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
Harsha,
Thanks so much for getting back with me. I will check the
logs, but I don't seem to get any error messages. I have a
nimbus AWS machine with zookeeper on it and a worker AWS
machine.
On the nimbus machine I start the zookeeper and then I run:
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui
On the worker machine I run:
bin/storm supervisor
When I go to the UI page, I only see 1 supervisor (the one on
the nimbus machine). So apparently, the worker machine isn't
"registering" with the nimbus machine.
On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[7...@harsha.io>
wrote:
Hi Stephen,
What are the issues you are seeing.
"How do worker machines "know" how to connect to nimbus? Is it
in the storm configuration file"
Yes. make sure you the supervisor(worker) , nimbus nodes are
able to connect to your zookeeper cluster.
Check your logs under storm_inst/logs/ for any errors when you
try to start nimbus or supervisors.
If you are installing it manually try following these steps if
you are not already done.
[8]http://www.michael-noll.com/tutorials/running-multi-node-sto
rm-cluster/
-Harsha
On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
All,
I would greatly appreciate any help that anyone would afford.
I've been trying to setup a storm cluster on AWS for a few
weeks now on centOS EC2 machines. So far, I haven't been able
to get a cluster built. I can get a supervisor and nimbus to
run on a single machine, but I can't figure out how to get
another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm
configuration file? I've gone through many tutorials and the
official documentation, but this point doesn't seem to be
covered anywhere in sufficient detail for a new guy like me.
Some of you may be tempted to point me toward storm-deploy,
but I spent four days trying to get that to work until I gave
up. I'm having Issue #58 on github. Following the instructions
exactly and other tutorials on a bran new AWS machine fails. So
I gave up on storm-deploy and decided to try and setup a
cluster manually. Thanks in advance to anyone willing to offer
me any inputs you can!
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual
or entity to which it is addressed and may contain information
that is confidential, privileged and exempt from disclosure
under applicable law. If the reader of this message is not the
intended recipient, you are hereby notified that any printing,
copying, dissemination, distribution, disclosure or forwarding
of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately and delete it from your system. Thank You.
References
1. http://54.68.149.181:8080/
2. mailto:hartzell.stephen@gmail.com
3. mailto:hartzell.stephen@gmail.com
4. mailto:pbrahmbhatt@hortonworks.com
5. mailto:storm@harsha.io
6. mailto:storm@harsha.io
7. mailto:storm@harsha.io
8. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
All,
I implemented the suggestions given by Parh and Harsha. I am now using
the default.yaml but I changed the storm.zookeeper.servers to the nimbus
machine's ip address: 54.68.149.181. I also changed the nimbus.host to
54.68.149.181. I also opened up port 6627. Now, the UI web page gives the
following error: org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused
You should be able to see the error it gives by going to the web page
yourself at: http://54.68.149.181:8080. I am only using this account to
test and see if I can even get storm to work, so these machines are only
for testing. Perhaps someone could tell me what the storm.yaml file should
look like for this setup?
-Thanks, Stephne
On Mon, Sep 8, 2014 at 7:41 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> I'm getting kind of confused by the storm.yaml file. Should I be using the
> default.yaml and just modify the zookeeper and nimbus ip, or should I use a
> bran new storm.yaml?
>
> My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on
> the nimbus machine. what should the storm.yaml look like on my worker and
> nimbus machine? Will the storm.yaml be the same on my worker and nimbus
> machine? I am not trying to do anything fancy, I am just trying to get a
> very basic cluster up and running.
>
> -Thanks, Stephen
>
> On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <
> hartzell.stephen@gmail.com> wrote:
>
>> All Thanks so much for your help. I cannot tell you how much I
>> appreciate it. I'm going to try out your suggestions and keep banging my
>> head again the wall : D. I've spent an enormous amount of time trying to
>> get this to work. I'll let you know what happens after I try to implement
>> your suggestions. It would be really cool if someone had a tutorial that
>> detailed this part. (I'll make it myself if I ever get this to work!) It
>> seems like trying to get a two-machine cluster setup on AWS would be a VERY
>> common use-case. I've read and watched everything I can on the topic and
>> nothing got it working for me!
>>
>> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>> The worker connects to the thrift port and not the ui port. You need to
>>> open port 6627 or whatever is the value being set in storm.yaml using
>>> property “nimbus.thrift.port”.
>>>
>>> Based on the configuration that you have pointed so far it seems your
>>> nimbus host has nimbus,ui,supervisor working because you actually have
>>> zookeeper running locally on that host. As Harsha pointed out you need to
>>> change it to a value that is the public ip instead of loopback interface.
>>>
>>> Thanks
>>> Parth
>>>
>>>
>>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>>
>>> storm.zookeeper.servers:
>>> - "127.0.0.1"
>>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>>> interface , instead either use your public ip or 0.0.0.0*)
>>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
>>> folder probably /home/storm, /tmp/storm will get deleted if your machine is
>>> restarted)*
>>>
>>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>>> 127.0.0.1.
>>>
>>> "No, I cannot ping my host which has a public ip address of
>>> 54.68.149.181"
>>> you are not able to reach this ip form worker node but able to access
>>> the UI using it?
>>> -Harsha
>>>
>>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>>
>>> Harsha,
>>>
>>> The storm.yaml on the host machine looks like this:
>>>
>>> storm.zookeeper.servers:
>>> - "127.0.0.1"
>>>
>>>
>>> nimbus.host: "127.0.0.1"
>>>
>>> storm.local.dir: /tmp/storm
>>>
>>>
>>> The storm.yaml on the worker machine looks like this:
>>>
>>> storm.zookeeper.servers:
>>> - "54.68.149.181"
>>>
>>>
>>> nimbus.host: "54.68.149.181"
>>>
>>> storm.local.dir: /tmp/storm
>>>
>>> No, I cannot ping my host which has a public ip address of 54.68.149.181
>>> although I can connect to the UI web page when it is hosted. I don't know
>>> how I would go about connecting to zookeeper on the nimbus host.
>>> -Thanks, Stephen
>>>
>>>
>>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>>
>>>
>>> There aren't any errors in worker machine supervisor logs. Are you using
>>> the same storm.yaml for both the machines and also are you able to ping
>>> your nimbus host or connect to zookeeper on nimbus host.
>>> -Harsha
>>>
>>>
>>>
>>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>>
>>> Harsha,
>>>
>>> Thanks so much for getting back with me. I will check the logs, but I
>>> don't seem to get any error messages. I have a nimbus AWS machine with
>>> zookeeper on it and a worker AWS machine.
>>>
>>> On the nimbus machine I start the zookeeper and then I run:
>>>
>>> bin/storm nimbus &
>>> bin/storm supervisor &
>>> bin/storm ui
>>>
>>> On the worker machine I run:
>>> bin/storm supervisor
>>>
>>> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
>>> machine). So apparently, the worker machine isn't "registering" with the
>>> nimbus machine.
>>>
>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>
>>>
>>> Hi Stephen,
>>> What are the issues you are seeing.
>>> "How do worker machines "know" how to connect to nimbus? Is it in the
>>> storm configuration file"
>>>
>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>>> connect to your zookeeper cluster.
>>> Check your logs under storm_inst/logs/ for any errors when you try to
>>> start nimbus or supervisors.
>>> If you are installing it manually try following these steps if you are
>>> not already done.
>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>> -Harsha
>>>
>>>
>>>
>>>
>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>
>>> All,
>>>
>>> I would greatly appreciate any help that anyone would afford. I've been
>>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
>>> machines. So far, I haven't been able to get a cluster built. I can get a
>>> supervisor and nimbus to run on a single machine, but I can't figure out
>>> how to get another worker to connect to nimbus. How do worker machines
>>> "know" how to connect to nimbus? Is it in the storm configuration file?
>>> I've gone through many tutorials and the official documentation, but this
>>> point doesn't seem to be covered anywhere in sufficient detail for a new
>>> guy like me.
>>>
>>> Some of you may be tempted to point me toward storm-deploy, but I
>>> spent four days trying to get that to work until I gave up. I'm having
>>> Issue #58 on github. Following the instructions exactly and other tutorials
>>> on a bran new AWS machine fails. So I gave up on storm-deploy and decided
>>> to try and setup a cluster manually. Thanks in advance to anyone willing to
>>> offer me any inputs you can!
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>
designing topologies
Posted by "Adaryl \"Bob\" Wakefield, MBA" <ad...@hotmail.com>.
When you design a topology do you still use MVC design patterns?
B.
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
I'm getting kind of confused by the storm.yaml file. Should I be using the
default.yaml and just modify the zookeeper and nimbus ip, or should I use a
bran new storm.yaml?
My nimbus machine has the ip address: 54.68.149.181. My zookeeper is on the
nimbus machine. what should the storm.yaml look like on my worker and
nimbus machine? Will the storm.yaml be the same on my worker and nimbus
machine? I am not trying to do anything fancy, I am just trying to get a
very basic cluster up and running.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 7:00 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> All Thanks so much for your help. I cannot tell you how much I
> appreciate it. I'm going to try out your suggestions and keep banging my
> head again the wall : D. I've spent an enormous amount of time trying to
> get this to work. I'll let you know what happens after I try to implement
> your suggestions. It would be really cool if someone had a tutorial that
> detailed this part. (I'll make it myself if I ever get this to work!) It
> seems like trying to get a two-machine cluster setup on AWS would be a VERY
> common use-case. I've read and watched everything I can on the topic and
> nothing got it working for me!
>
> On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> The worker connects to the thrift port and not the ui port. You need to
>> open port 6627 or whatever is the value being set in storm.yaml using
>> property “nimbus.thrift.port”.
>>
>> Based on the configuration that you have pointed so far it seems your
>> nimbus host has nimbus,ui,supervisor working because you actually have
>> zookeeper running locally on that host. As Harsha pointed out you need to
>> change it to a value that is the public ip instead of loopback interface.
>>
>> Thanks
>> Parth
>>
>>
>> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>>
>> storm.zookeeper.servers:
>> - "127.0.0.1"
>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>> interface , instead either use your public ip or 0.0.0.0*)
>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
>> folder probably /home/storm, /tmp/storm will get deleted if your machine is
>> restarted)*
>>
>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>> 127.0.0.1.
>>
>> "No, I cannot ping my host which has a public ip address of 54.68.149.181"
>> you are not able to reach this ip form worker node but able to access the
>> UI using it?
>> -Harsha
>>
>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>
>> Harsha,
>>
>> The storm.yaml on the host machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "127.0.0.1"
>>
>>
>> nimbus.host: "127.0.0.1"
>>
>> storm.local.dir: /tmp/storm
>>
>>
>> The storm.yaml on the worker machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "54.68.149.181"
>>
>>
>> nimbus.host: "54.68.149.181"
>>
>> storm.local.dir: /tmp/storm
>>
>> No, I cannot ping my host which has a public ip address of 54.68.149.181
>> although I can connect to the UI web page when it is hosted. I don't know
>> how I would go about connecting to zookeeper on the nimbus host.
>> -Thanks, Stephen
>>
>>
>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>
>>
>> There aren't any errors in worker machine supervisor logs. Are you using
>> the same storm.yaml for both the machines and also are you able to ping
>> your nimbus host or connect to zookeeper on nimbus host.
>> -Harsha
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>
>> Harsha,
>>
>> Thanks so much for getting back with me. I will check the logs, but I
>> don't seem to get any error messages. I have a nimbus AWS machine with
>> zookeeper on it and a worker AWS machine.
>>
>> On the nimbus machine I start the zookeeper and then I run:
>>
>> bin/storm nimbus &
>> bin/storm supervisor &
>> bin/storm ui
>>
>> On the worker machine I run:
>> bin/storm supervisor
>>
>> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
>> machine). So apparently, the worker machine isn't "registering" with the
>> nimbus machine.
>>
>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>
>>
>> Hi Stephen,
>> What are the issues you are seeing.
>> "How do worker machines "know" how to connect to nimbus? Is it in the
>> storm configuration file"
>>
>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>> connect to your zookeeper cluster.
>> Check your logs under storm_inst/logs/ for any errors when you try to
>> start nimbus or supervisors.
>> If you are installing it manually try following these steps if you are
>> not already done.
>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>> -Harsha
>>
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>
>> All,
>>
>> I would greatly appreciate any help that anyone would afford. I've been
>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
>> machines. So far, I haven't been able to get a cluster built. I can get a
>> supervisor and nimbus to run on a single machine, but I can't figure out
>> how to get another worker to connect to nimbus. How do worker machines
>> "know" how to connect to nimbus? Is it in the storm configuration file?
>> I've gone through many tutorials and the official documentation, but this
>> point doesn't seem to be covered anywhere in sufficient detail for a new
>> guy like me.
>>
>> Some of you may be tempted to point me toward storm-deploy, but I spent
>> four days trying to get that to work until I gave up. I'm having Issue #58
>> on github. Following the instructions exactly and other tutorials on a bran
>> new AWS machine fails. So I gave up on storm-deploy and decided to try and
>> setup a cluster manually. Thanks in advance to anyone willing to offer me
>> any inputs you can!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
All Thanks so much for your help. I cannot tell you how much I appreciate
it. I'm going to try out your suggestions and keep banging my head again
the wall : D. I've spent an enormous amount of time trying to get this to
work. I'll let you know what happens after I try to implement your
suggestions. It would be really cool if someone had a tutorial that
detailed this part. (I'll make it myself if I ever get this to work!) It
seems like trying to get a two-machine cluster setup on AWS would be a VERY
common use-case. I've read and watched everything I can on the topic and
nothing got it working for me!
On Mon, Sep 8, 2014 at 6:54 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> The worker connects to the thrift port and not the ui port. You need to
> open port 6627 or whatever is the value being set in storm.yaml using
> property “nimbus.thrift.port”.
>
> Based on the configuration that you have pointed so far it seems your
> nimbus host has nimbus,ui,supervisor working because you actually have
> zookeeper running locally on that host. As Harsha pointed out you need to
> change it to a value that is the public ip instead of loopback interface.
>
> Thanks
> Parth
>
>
> On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
>
> storm.zookeeper.servers:
> - "127.0.0.1"
> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback interface
> , instead either use your public ip or 0.0.0.0*)
> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
> folder probably /home/storm, /tmp/storm will get deleted if your machine is
> restarted)*
>
> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
> 127.0.0.1.
>
> "No, I cannot ping my host which has a public ip address of 54.68.149.181"
> you are not able to reach this ip form worker node but able to access the
> UI using it?
> -Harsha
>
> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>
> Harsha,
>
> The storm.yaml on the host machine looks like this:
>
> storm.zookeeper.servers:
> - "127.0.0.1"
>
>
> nimbus.host: "127.0.0.1"
>
> storm.local.dir: /tmp/storm
>
>
> The storm.yaml on the worker machine looks like this:
>
> storm.zookeeper.servers:
> - "54.68.149.181"
>
>
> nimbus.host: "54.68.149.181"
>
> storm.local.dir: /tmp/storm
>
> No, I cannot ping my host which has a public ip address of 54.68.149.181
> although I can connect to the UI web page when it is hosted. I don't know
> how I would go about connecting to zookeeper on the nimbus host.
> -Thanks, Stephen
>
>
> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>
>
> There aren't any errors in worker machine supervisor logs. Are you using
> the same storm.yaml for both the machines and also are you able to ping
> your nimbus host or connect to zookeeper on nimbus host.
> -Harsha
>
>
>
> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>
> Harsha,
>
> Thanks so much for getting back with me. I will check the logs, but I
> don't seem to get any error messages. I have a nimbus AWS machine with
> zookeeper on it and a worker AWS machine.
>
> On the nimbus machine I start the zookeeper and then I run:
>
> bin/storm nimbus &
> bin/storm supervisor &
> bin/storm ui
>
> On the worker machine I run:
> bin/storm supervisor
>
> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
> machine). So apparently, the worker machine isn't "registering" with the
> nimbus machine.
>
> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi Stephen,
> What are the issues you are seeing.
> "How do worker machines "know" how to connect to nimbus? Is it in the
> storm configuration file"
>
> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
> connect to your zookeeper cluster.
> Check your logs under storm_inst/logs/ for any errors when you try to
> start nimbus or supervisors.
> If you are installing it manually try following these steps if you are not
> already done.
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
> -Harsha
>
>
>
>
> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>
> All,
>
> I would greatly appreciate any help that anyone would afford. I've been
> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
> machines. So far, I haven't been able to get a cluster built. I can get a
> supervisor and nimbus to run on a single machine, but I can't figure out
> how to get another worker to connect to nimbus. How do worker machines
> "know" how to connect to nimbus? Is it in the storm configuration file?
> I've gone through many tutorials and the official documentation, but this
> point doesn't seem to be covered anywhere in sufficient detail for a new
> guy like me.
>
> Some of you may be tempted to point me toward storm-deploy, but I spent
> four days trying to get that to work until I gave up. I'm having Issue #58
> on github. Following the instructions exactly and other tutorials on a bran
> new AWS machine fails. So I gave up on storm-deploy and decided to try and
> setup a cluster manually. Thanks in advance to anyone willing to offer me
> any inputs you can!
>
>
>
>
>
>
>
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: How Do Workers Connect To Nimbus
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
The worker connects to the thrift port and not the ui port. You need to open port 6627 or whatever is the value being set in storm.yaml using property “nimbus.thrift.port”.
Based on the configuration that you have pointed so far it seems your nimbus host has nimbus,ui,supervisor working because you actually have zookeeper running locally on that host. As Harsha pointed out you need to change it to a value that is the public ip instead of loopback interface.
Thanks
Parth
On Sep 8, 2014, at 3:42 PM, Harsha <st...@harsha.io> wrote:
> storm.zookeeper.servers:
> - "127.0.0.1"
> nimbus.host: "127.0.0.1" ( 127.0.0.1 causes to bind a loopback interface , instead either use your public ip or 0.0.0.0)
> storm.local.dir: /tmp/storm ( I recommend this to move to a different folder probably /home/storm, /tmp/storm will get deleted if your machine is restarted)
>
> make sure you zookeeper is also listening in 0.0.0.0 or public ip not 127.0.0.1.
>
> "No, I cannot ping my host which has a public ip address of 54.68.149.181"
> you are not able to reach this ip form worker node but able to access the UI using it?
> -Harsha
>
> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>> Harsha,
>>
>> The storm.yaml on the host machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "127.0.0.1"
>>
>>
>> nimbus.host: "127.0.0.1"
>>
>> storm.local.dir: /tmp/storm
>>
>>
>> The storm.yaml on the worker machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "54.68.149.181"
>>
>>
>> nimbus.host: "54.68.149.181"
>>
>> storm.local.dir: /tmp/storm
>>
>> No, I cannot ping my host which has a public ip address of 54.68.149.181 although I can connect to the UI web page when it is hosted. I don't know how I would go about connecting to zookeeper on the nimbus host.
>> -Thanks, Stephen
>>
>>
>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>
>> There aren't any errors in worker machine supervisor logs. Are you using the same storm.yaml for both the machines and also are you able to ping your nimbus host or connect to zookeeper on nimbus host.
>> -Harsha
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>> Harsha,
>>>
>>> Thanks so much for getting back with me. I will check the logs, but I don't seem to get any error messages. I have a nimbus AWS machine with zookeeper on it and a worker AWS machine.
>>>
>>> On the nimbus machine I start the zookeeper and then I run:
>>>
>>> bin/storm nimbus &
>>> bin/storm supervisor &
>>> bin/storm ui
>>>
>>> On the worker machine I run:
>>> bin/storm supervisor
>>>
>>> When I go to the UI page, I only see 1 supervisor (the one on the nimbus machine). So apparently, the worker machine isn't "registering" with the nimbus machine.
>>>
>>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>>
>>> Hi Stephen,
>>> What are the issues you are seeing.
>>> "How do worker machines "know" how to connect to nimbus? Is it in the storm configuration file"
>>>
>>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to connect to your zookeeper cluster.
>>> Check your logs under storm_inst/logs/ for any errors when you try to start nimbus or supervisors.
>>> If you are installing it manually try following these steps if you are not already done.
>>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>>> -Harsha
>>>
>>>
>>>
>>>
>>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>>> All,
>>>>
>>>> I would greatly appreciate any help that anyone would afford. I've been trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 machines. So far, I haven't been able to get a cluster built. I can get a supervisor and nimbus to run on a single machine, but I can't figure out how to get another worker to connect to nimbus. How do worker machines "know" how to connect to nimbus? Is it in the storm configuration file? I've gone through many tutorials and the official documentation, but this point doesn't seem to be covered anywhere in sufficient detail for a new guy like me.
>>>>
>>>> Some of you may be tempted to point me toward storm-deploy, but I spent four days trying to get that to work until I gave up. I'm having Issue #58 on github. Following the instructions exactly and other tutorials on a bran new AWS machine fails. So I gave up on storm-deploy and decided to try and setup a cluster manually. Thanks in advance to anyone willing to offer me any inputs you can!
>>>
>>>
>>
>>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
I am accesing http://54.68.149.181:8080/ using chrome. But somehow ping
doesn't work says "could not find host http://54.68.149.181:8080/"
On Mon, Sep 8, 2014 at 6:49 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> I'll go ahead and implement your suggestion for the .yaml.
>
> Your other question is also confusing to me. I can access the storm UI
> from my personal computer using http://54.68.149.181:8080/, but my
> personal computer cannot ping it nor can my worker AWS machine.
>
> On Mon, Sep 8, 2014 at 6:42 PM, Harsha <st...@harsha.io> wrote:
>
>> storm.zookeeper.servers:
>> - "127.0.0.1"
>> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback
>> interface , instead either use your public ip or 0.0.0.0*)
>> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
>> folder probably /home/storm, /tmp/storm will get deleted if your machine is
>> restarted)*
>>
>> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
>> 127.0.0.1.
>>
>> "No, I cannot ping my host which has a public ip address of 54.68.149.181"
>> you are not able to reach this ip form worker node but able to access the
>> UI using it?
>> -Harsha
>>
>> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>>
>> Harsha,
>>
>> The storm.yaml on the host machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "127.0.0.1"
>>
>>
>> nimbus.host: "127.0.0.1"
>>
>> storm.local.dir: /tmp/storm
>>
>>
>> The storm.yaml on the worker machine looks like this:
>>
>> storm.zookeeper.servers:
>> - "54.68.149.181"
>>
>>
>> nimbus.host: "54.68.149.181"
>>
>> storm.local.dir: /tmp/storm
>>
>> No, I cannot ping my host which has a public ip address of 54.68.149.181
>> although I can connect to the UI web page when it is hosted. I don't know
>> how I would go about connecting to zookeeper on the nimbus host.
>> -Thanks, Stephen
>>
>>
>> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>>
>>
>> There aren't any errors in worker machine supervisor logs. Are you using
>> the same storm.yaml for both the machines and also are you able to ping
>> your nimbus host or connect to zookeeper on nimbus host.
>> -Harsha
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>>
>> Harsha,
>>
>> Thanks so much for getting back with me. I will check the logs, but I
>> don't seem to get any error messages. I have a nimbus AWS machine with
>> zookeeper on it and a worker AWS machine.
>>
>> On the nimbus machine I start the zookeeper and then I run:
>>
>> bin/storm nimbus &
>> bin/storm supervisor &
>> bin/storm ui
>>
>> On the worker machine I run:
>> bin/storm supervisor
>>
>> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
>> machine). So apparently, the worker machine isn't "registering" with the
>> nimbus machine.
>>
>> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>>
>>
>> Hi Stephen,
>> What are the issues you are seeing.
>> "How do worker machines "know" how to connect to nimbus? Is it in the
>> storm configuration file"
>>
>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>> connect to your zookeeper cluster.
>> Check your logs under storm_inst/logs/ for any errors when you try to
>> start nimbus or supervisors.
>> If you are installing it manually try following these steps if you are
>> not already done.
>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>> -Harsha
>>
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>
>> All,
>>
>> I would greatly appreciate any help that anyone would afford. I've been
>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
>> machines. So far, I haven't been able to get a cluster built. I can get a
>> supervisor and nimbus to run on a single machine, but I can't figure out
>> how to get another worker to connect to nimbus. How do worker machines
>> "know" how to connect to nimbus? Is it in the storm configuration file?
>> I've gone through many tutorials and the official documentation, but this
>> point doesn't seem to be covered anywhere in sufficient detail for a new
>> guy like me.
>>
>> Some of you may be tempted to point me toward storm-deploy, but I spent
>> four days trying to get that to work until I gave up. I'm having Issue #58
>> on github. Following the instructions exactly and other tutorials on a bran
>> new AWS machine fails. So I gave up on storm-deploy and decided to try and
>> setup a cluster manually. Thanks in advance to anyone willing to offer me
>> any inputs you can!
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
I'll go ahead and implement your suggestion for the .yaml.
Your other question is also confusing to me. I can access the storm UI from
my personal computer using http://54.68.149.181:8080/, but my personal
computer cannot ping it nor can my worker AWS machine.
On Mon, Sep 8, 2014 at 6:42 PM, Harsha <st...@harsha.io> wrote:
> storm.zookeeper.servers:
> - "127.0.0.1"
> nimbus.host: "127.0.0.1" ( *127.0.0.1 causes to bind a loopback interface
> , instead either use your public ip or 0.0.0.0*)
> storm.local.dir: /tmp/storm ( I* recommend this to move to a different
> folder probably /home/storm, /tmp/storm will get deleted if your machine is
> restarted)*
>
> make sure you zookeeper is also listening in 0.0.0.0 or public ip not
> 127.0.0.1.
>
> "No, I cannot ping my host which has a public ip address of 54.68.149.181"
> you are not able to reach this ip form worker node but able to access the
> UI using it?
> -Harsha
>
> On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
>
> Harsha,
>
> The storm.yaml on the host machine looks like this:
>
> storm.zookeeper.servers:
> - "127.0.0.1"
>
>
> nimbus.host: "127.0.0.1"
>
> storm.local.dir: /tmp/storm
>
>
> The storm.yaml on the worker machine looks like this:
>
> storm.zookeeper.servers:
> - "54.68.149.181"
>
>
> nimbus.host: "54.68.149.181"
>
> storm.local.dir: /tmp/storm
>
> No, I cannot ping my host which has a public ip address of 54.68.149.181
> although I can connect to the UI web page when it is hosted. I don't know
> how I would go about connecting to zookeeper on the nimbus host.
> -Thanks, Stephen
>
>
> On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
>
>
> There aren't any errors in worker machine supervisor logs. Are you using
> the same storm.yaml for both the machines and also are you able to ping
> your nimbus host or connect to zookeeper on nimbus host.
> -Harsha
>
>
>
> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>
> Harsha,
>
> Thanks so much for getting back with me. I will check the logs, but I
> don't seem to get any error messages. I have a nimbus AWS machine with
> zookeeper on it and a worker AWS machine.
>
> On the nimbus machine I start the zookeeper and then I run:
>
> bin/storm nimbus &
> bin/storm supervisor &
> bin/storm ui
>
> On the worker machine I run:
> bin/storm supervisor
>
> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
> machine). So apparently, the worker machine isn't "registering" with the
> nimbus machine.
>
> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi Stephen,
> What are the issues you are seeing.
> "How do worker machines "know" how to connect to nimbus? Is it in the
> storm configuration file"
>
> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
> connect to your zookeeper cluster.
> Check your logs under storm_inst/logs/ for any errors when you try to
> start nimbus or supervisors.
> If you are installing it manually try following these steps if you are not
> already done.
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
> -Harsha
>
>
>
>
> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>
> All,
>
> I would greatly appreciate any help that anyone would afford. I've been
> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
> machines. So far, I haven't been able to get a cluster built. I can get a
> supervisor and nimbus to run on a single machine, but I can't figure out
> how to get another worker to connect to nimbus. How do worker machines
> "know" how to connect to nimbus? Is it in the storm configuration file?
> I've gone through many tutorials and the official documentation, but this
> point doesn't seem to be covered anywhere in sufficient detail for a new
> guy like me.
>
> Some of you may be tempted to point me toward storm-deploy, but I spent
> four days trying to get that to work until I gave up. I'm having Issue #58
> on github. Following the instructions exactly and other tutorials on a bran
> new AWS machine fails. So I gave up on storm-deploy and decided to try and
> setup a cluster manually. Thanks in advance to anyone willing to offer me
> any inputs you can!
>
>
>
>
>
>
>
>
>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Harsha <st...@harsha.io>.
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1" ( 127.0.0.1 causes to bind a loopback
interface , instead either use your public ip or 0.0.0.0)
storm.local.dir: /tmp/storm ( I recommend this to move to a
different folder probably /home/storm, /tmp/storm will get
deleted if your machine is restarted)
make sure you zookeeper is also listening in 0.0.0.0 or public
ip not 127.0.0.1.
"No, I cannot ping my host which has a public ip address of
54.68.149.181"
you are not able to reach this ip form worker node but able to
access the UI using it?
-Harsha
On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote:
Harsha,
The storm.yaml on the host machine looks like this:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1"
storm.local.dir: /tmp/storm
The storm.yaml on the worker machine looks like this:
storm.zookeeper.servers:
- "54.68.149.181"
nimbus.host: "54.68.149.181"
storm.local.dir: /tmp/storm
No, I cannot ping my host which has a public ip address of
54.68.149.181 although I can connect to the UI web page when it
is hosted. I don't know how I would go about connecting to
zookeeper on the nimbus host.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 6:28 PM, Harsha <[1...@harsha.io>
wrote:
There aren't any errors in worker machine supervisor logs. Are
you using the same storm.yaml for both the machines and also
are you able to ping your nimbus host or connect to zookeeper
on nimbus host.
-Harsha
On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
Harsha,
Thanks so much for getting back with me. I will check the
logs, but I don't seem to get any error messages. I have a
nimbus AWS machine with zookeeper on it and a worker AWS
machine.
On the nimbus machine I start the zookeeper and then I run:
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui
On the worker machine I run:
bin/storm supervisor
When I go to the UI page, I only see 1 supervisor (the one on
the nimbus machine). So apparently, the worker machine isn't
"registering" with the nimbus machine.
On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[2...@harsha.io>
wrote:
Hi Stephen,
What are the issues you are seeing.
"How do worker machines "know" how to connect to nimbus? Is it
in the storm configuration file"
Yes. make sure you the supervisor(worker) , nimbus nodes are
able to connect to your zookeeper cluster.
Check your logs under storm_inst/logs/ for any errors when you
try to start nimbus or supervisors.
If you are installing it manually try following these steps if
you are not already done.
[3]http://www.michael-noll.com/tutorials/running-multi-node-sto
rm-cluster/
-Harsha
On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
All,
I would greatly appreciate any help that anyone would afford.
I've been trying to setup a storm cluster on AWS for a few
weeks now on centOS EC2 machines. So far, I haven't been able
to get a cluster built. I can get a supervisor and nimbus to
run on a single machine, but I can't figure out how to get
another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm
configuration file? I've gone through many tutorials and the
official documentation, but this point doesn't seem to be
covered anywhere in sufficient detail for a new guy like me.
Some of you may be tempted to point me toward storm-deploy,
but I spent four days trying to get that to work until I gave
up. I'm having Issue #58 on github. Following the instructions
exactly and other tutorials on a bran new AWS machine fails. So
I gave up on storm-deploy and decided to try and setup a
cluster manually. Thanks in advance to anyone willing to offer
me any inputs you can!
References
1. mailto:storm@harsha.io
2. mailto:storm@harsha.io
3. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
Also, to answer Harsha's question. I have no errors in any of my logs on
the nimbus or worker machines.
On Mon, Sep 8, 2014 at 6:39 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> I opened up port 8080 so that I could access the UI page. Do I need to
> open up something beyond that?
>
> On Mon, Sep 8, 2014 at 6:37 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> One thing that is generally needed on EC2 hosts is security group setting
>>
>>
>> http://docs.aws.amazon.com/gettingstarted/latest/computebasics/getting-started-security-group.html
>>
>> If you can not ping the host I am guessing this might be one of the
>> things that you need to setup correctly.
>>
>> On Sep 8, 2014, at 3:34 PM, Stephen Hartzell <ha...@gmail.com>
>> wrote:
>>
>> cannot
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
I opened up port 8080 so that I could access the UI page. Do I need to open
up something beyond that?
On Mon, Sep 8, 2014 at 6:37 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:
> One thing that is generally needed on EC2 hosts is security group setting
>
>
> http://docs.aws.amazon.com/gettingstarted/latest/computebasics/getting-started-security-group.html
>
> If you can not ping the host I am guessing this might be one of the things
> that you need to setup correctly.
>
> On Sep 8, 2014, at 3:34 PM, Stephen Hartzell <ha...@gmail.com>
> wrote:
>
> cannot
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: How Do Workers Connect To Nimbus
Posted by Parth Brahmbhatt <pb...@hortonworks.com>.
One thing that is generally needed on EC2 hosts is security group setting
http://docs.aws.amazon.com/gettingstarted/latest/computebasics/getting-started-security-group.html
If you can not ping the host I am guessing this might be one of the things that you need to setup correctly.
On Sep 8, 2014, at 3:34 PM, Stephen Hartzell <ha...@gmail.com> wrote:
> cannot
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
Harsha,
The storm.yaml on the host machine looks like this:
storm.zookeeper.servers:
- "127.0.0.1"
nimbus.host: "127.0.0.1"
storm.local.dir: /tmp/storm
The storm.yaml on the worker machine looks like this:
storm.zookeeper.servers:
- "54.68.149.181"
nimbus.host: "54.68.149.181"
storm.local.dir: /tmp/storm
No, I cannot ping my host which has a public ip address of 54.68.149.181
although I can connect to the UI web page when it is hosted. I don't know
how I would go about connecting to zookeeper on the nimbus host.
-Thanks, Stephen
On Mon, Sep 8, 2014 at 6:28 PM, Harsha <st...@harsha.io> wrote:
> There aren't any errors in worker machine supervisor logs. Are you using
> the same storm.yaml for both the machines and also are you able to ping
> your nimbus host or connect to zookeeper on nimbus host.
> -Harsha
>
>
> On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
>
> Harsha,
>
> Thanks so much for getting back with me. I will check the logs, but I
> don't seem to get any error messages. I have a nimbus AWS machine with
> zookeeper on it and a worker AWS machine.
>
> On the nimbus machine I start the zookeeper and then I run:
>
> bin/storm nimbus &
> bin/storm supervisor &
> bin/storm ui
>
> On the worker machine I run:
> bin/storm supervisor
>
> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
> machine). So apparently, the worker machine isn't "registering" with the
> nimbus machine.
>
> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>
>
> Hi Stephen,
> What are the issues you are seeing.
> "How do worker machines "know" how to connect to nimbus? Is it in the
> storm configuration file"
>
> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
> connect to your zookeeper cluster.
> Check your logs under storm_inst/logs/ for any errors when you try to
> start nimbus or supervisors.
> If you are installing it manually try following these steps if you are not
> already done.
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
> -Harsha
>
>
>
>
> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>
> All,
>
> I would greatly appreciate any help that anyone would afford. I've been
> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
> machines. So far, I haven't been able to get a cluster built. I can get a
> supervisor and nimbus to run on a single machine, but I can't figure out
> how to get another worker to connect to nimbus. How do worker machines
> "know" how to connect to nimbus? Is it in the storm configuration file?
> I've gone through many tutorials and the official documentation, but this
> point doesn't seem to be covered anywhere in sufficient detail for a new
> guy like me.
>
> Some of you may be tempted to point me toward storm-deploy, but I spent
> four days trying to get that to work until I gave up. I'm having Issue #58
> on github. Following the instructions exactly and other tutorials on a bran
> new AWS machine fails. So I gave up on storm-deploy and decided to try and
> setup a cluster manually. Thanks in advance to anyone willing to offer me
> any inputs you can!
>
>
>
>
>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Harsha <st...@harsha.io>.
There aren't any errors in worker machine supervisor logs. Are
you using the same storm.yaml for both the machines and also
are you able to ping your nimbus host or connect to zookeeper
on nimbus host.
-Harsha
On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote:
Harsha,
Thanks so much for getting back with me. I will check the
logs, but I don't seem to get any error messages. I have a
nimbus AWS machine with zookeeper on it and a worker AWS
machine.
On the nimbus machine I start the zookeeper and then I run:
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui
On the worker machine I run:
bin/storm supervisor
When I go to the UI page, I only see 1 supervisor (the one on
the nimbus machine). So apparently, the worker machine isn't
"registering" with the nimbus machine.
On Mon, Sep 8, 2014 at 6:16 PM, Harsha <[1...@harsha.io>
wrote:
Hi Stephen,
What are the issues you are seeing.
"How do worker machines "know" how to connect to nimbus? Is it
in the storm configuration file"
Yes. make sure you the supervisor(worker) , nimbus nodes are
able to connect to your zookeeper cluster.
Check your logs under storm_inst/logs/ for any errors when you
try to start nimbus or supervisors.
If you are installing it manually try following these steps if
you are not already done.
[2]http://www.michael-noll.com/tutorials/running-multi-node-sto
rm-cluster/
-Harsha
On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
All,
I would greatly appreciate any help that anyone would afford.
I've been trying to setup a storm cluster on AWS for a few
weeks now on centOS EC2 machines. So far, I haven't been able
to get a cluster built. I can get a supervisor and nimbus to
run on a single machine, but I can't figure out how to get
another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm
configuration file? I've gone through many tutorials and the
official documentation, but this point doesn't seem to be
covered anywhere in sufficient detail for a new guy like me.
Some of you may be tempted to point me toward storm-deploy,
but I spent four days trying to get that to work until I gave
up. I'm having Issue #58 on github. Following the instructions
exactly and other tutorials on a bran new AWS machine fails. So
I gave up on storm-deploy and decided to try and setup a
cluster manually. Thanks in advance to anyone willing to offer
me any inputs you can!
References
1. mailto:storm@harsha.io
2. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
Also, one of the first things that I had tried to do is follow that
tutorial. The only problem is that tutorial appears to gloss over the setup
of the worker nodes. It just says that they will automatically register
with zookeeper. But there must be some setup or configuration to tell the
machine where the zookeeper machine is! The tutorial doesn't seem to
explain that part.
On Mon, Sep 8, 2014 at 6:24 PM, Stephen Hartzell <hartzell.stephen@gmail.com
> wrote:
> Harsha,
>
> Thanks so much for getting back with me. I will check the logs, but I
> don't seem to get any error messages. I have a nimbus AWS machine with
> zookeeper on it and a worker AWS machine.
>
> On the nimbus machine I start the zookeeper and then I run:
>
> bin/storm nimbus &
> bin/storm supervisor &
> bin/storm ui
>
> On the worker machine I run:
> bin/storm supervisor
>
> When I go to the UI page, I only see 1 supervisor (the one on the nimbus
> machine). So apparently, the worker machine isn't "registering" with the
> nimbus machine.
>
> On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
>
>> Hi Stephen,
>> What are the issues you are seeing.
>> "How do worker machines "know" how to connect to nimbus? Is it in the
>> storm configuration file"
>> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
>> connect to your zookeeper cluster.
>> Check your logs under storm_inst/logs/ for any errors when you try to
>> start nimbus or supervisors.
>> If you are installing it manually try following these steps if you are
>> not already done.
>> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
>> -Harsha
>>
>>
>>
>> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>>
>> All,
>>
>> I would greatly appreciate any help that anyone would afford. I've been
>> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
>> machines. So far, I haven't been able to get a cluster built. I can get a
>> supervisor and nimbus to run on a single machine, but I can't figure out
>> how to get another worker to connect to nimbus. How do worker machines
>> "know" how to connect to nimbus? Is it in the storm configuration file?
>> I've gone through many tutorials and the official documentation, but this
>> point doesn't seem to be covered anywhere in sufficient detail for a new
>> guy like me.
>>
>> Some of you may be tempted to point me toward storm-deploy, but I spent
>> four days trying to get that to work until I gave up. I'm having Issue #58
>> on github. Following the instructions exactly and other tutorials on a bran
>> new AWS machine fails. So I gave up on storm-deploy and decided to try and
>> setup a cluster manually. Thanks in advance to anyone willing to offer me
>> any inputs you can!
>>
>>
>>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Stephen Hartzell <ha...@gmail.com>.
Harsha,
Thanks so much for getting back with me. I will check the logs, but I
don't seem to get any error messages. I have a nimbus AWS machine with
zookeeper on it and a worker AWS machine.
On the nimbus machine I start the zookeeper and then I run:
bin/storm nimbus &
bin/storm supervisor &
bin/storm ui
On the worker machine I run:
bin/storm supervisor
When I go to the UI page, I only see 1 supervisor (the one on the nimbus
machine). So apparently, the worker machine isn't "registering" with the
nimbus machine.
On Mon, Sep 8, 2014 at 6:16 PM, Harsha <st...@harsha.io> wrote:
> Hi Stephen,
> What are the issues you are seeing.
> "How do worker machines "know" how to connect to nimbus? Is it in the
> storm configuration file"
> Yes. make sure you the supervisor(worker) , nimbus nodes are able to
> connect to your zookeeper cluster.
> Check your logs under storm_inst/logs/ for any errors when you try to
> start nimbus or supervisors.
> If you are installing it manually try following these steps if you are not
> already done.
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/
> -Harsha
>
>
>
> On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
>
> All,
>
> I would greatly appreciate any help that anyone would afford. I've been
> trying to setup a storm cluster on AWS for a few weeks now on centOS EC2
> machines. So far, I haven't been able to get a cluster built. I can get a
> supervisor and nimbus to run on a single machine, but I can't figure out
> how to get another worker to connect to nimbus. How do worker machines
> "know" how to connect to nimbus? Is it in the storm configuration file?
> I've gone through many tutorials and the official documentation, but this
> point doesn't seem to be covered anywhere in sufficient detail for a new
> guy like me.
>
> Some of you may be tempted to point me toward storm-deploy, but I spent
> four days trying to get that to work until I gave up. I'm having Issue #58
> on github. Following the instructions exactly and other tutorials on a bran
> new AWS machine fails. So I gave up on storm-deploy and decided to try and
> setup a cluster manually. Thanks in advance to anyone willing to offer me
> any inputs you can!
>
>
>
Re: How Do Workers Connect To Nimbus
Posted by Harsha <st...@harsha.io>.
Hi Stephen,
What are the issues you are seeing.
"How do worker machines "know" how to connect to nimbus? Is it
in the storm configuration file"
Yes. make sure you the supervisor(worker) , nimbus nodes are
able to connect to your zookeeper cluster.
Check your logs under storm_inst/logs/ for any errors when you
try to start nimbus or supervisors.
If you are installing it manually try following these steps if
you are not already done.
[1]http://www.michael-noll.com/tutorials/running-multi-node-sto
rm-cluster/
-Harsha
On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote:
All,
I would greatly appreciate any help that anyone would afford.
I've been trying to setup a storm cluster on AWS for a few
weeks now on centOS EC2 machines. So far, I haven't been able
to get a cluster built. I can get a supervisor and nimbus to
run on a single machine, but I can't figure out how to get
another worker to connect to nimbus. How do worker machines
"know" how to connect to nimbus? Is it in the storm
configuration file? I've gone through many tutorials and the
official documentation, but this point doesn't seem to be
covered anywhere in sufficient detail for a new guy like me.
Some of you may be tempted to point me toward storm-deploy,
but I spent four days trying to get that to work until I gave
up. I'm having Issue #58 on github. Following the instructions
exactly and other tutorials on a bran new AWS machine fails. So
I gave up on storm-deploy and decided to try and setup a
cluster manually. Thanks in advance to anyone willing to offer
me any inputs you can!
References
1. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/