You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by sam mohel <sa...@gmail.com> on 2015/12/09 19:19:40 UTC

Problem with storm since 4 months

I have this problem since 4months when I submitted topology I got this in
the worker log file [ERROR] Async loop died! org.zeromq.ZMQException:
Address already in use(0x62)
at org.zeromq.ZMQ$Socket.bind(Native Method)
at zilch.mq$bind.invoke(mq.clj:69)
at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown Source)

when i tried to connect port 6703 and 6702

And supervisor log file hadn't still start


I searched everywhere but cannot find any solution I hope you can

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
thanks for replying
for
point 1. the local port range of my machine default is 32768    61000 and i
changed it to 1024    65535 is that right or not ?
point 2. how can i checked it , please
point 3. also how can i know that , it may be the problem

really Thanks for your time

On Thu, Dec 10, 2015 at 5:37 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> You can try to find which process has bound this port by "netstat -anp"
> first.
>
> Generally, there are following cases for the binding error.
> 1. "local port range" is not set to exclude the port range used in Storm.
> 2. The previous worker was not killed correctly.
> 3. There is bug of assignment in some scenarios. Same port was assigned to
> two workers.
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Thursday, December 10, 2015 7:16 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> i tried to use storm-0.9.5 but problem changed with
>
> cannot bind port 6703 i think it's same problem
>
> On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
>
> > Sam,
> >           you might be using very old version of storm since its showing
> >           ZeroMQ. Can you try using newer version storm without zero mq.
> > -Harsha
> >
> > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > I have this problem since 4months when I submitted topology I got
> > > this in the worker log file [ERROR] Async loop died!
> org.zeromq.ZMQException:
> > > Address already in use(0x62)
> > > at org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > zilch.mq$bind.invoke(mq.clj:69) at
> > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > >
> > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> > nvoke(loader.clj:26)
> > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> > > Source)
> > >
> > > when i tried to connect port 6703 and 6702
> > >
> > > And supervisor log file hadn't still start
> > >
> > >
> > > I searched everywhere but cannot find any solution I hope you can
> >
>
>

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
thanks but i should run it with -0.8.2 however that i updgraded it before
and got that it couldn't bind port i need to try that

It was indeed a port conflict, but not
with another ZMQ process. It turns out our ephemeral port range was messed
up on the machines:


$ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000


but the defuakt i have is 32768    61000


but don't know what are the right numbers ?

On Thu, Dec 17, 2015 at 12:03 PM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> Not sure what the reason is why ZMQ cause this error.
> If you are using a new version Storm(e.g 0.9.x), could you try to use
> Netty for message transport, since ZMQ is no longer used in Storm now?
> Please make sure following configuration is correct.
> - storm.messaging.transport: "backtype.storm.messaging.netty.Context"
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Thursday, December 17, 2015 5:08 PM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> i edited to make topology.debug true and got in the supervisor log file
> still hasn't start and in the worker log file
>
> 2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"]
> 2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33]
> 2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33)
> 2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33]
> 2015-12-17 07:52:23 worker [INFO] Launching receive-thread for
> 5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706
> 2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33)
>
> after finished loading got alot of this lines
>
> 2015-12-17 07:52:27 executor [INFO] Processing received message
> source: __system:-1, stream: __tick, id: {}, [5]
> 2015-12-17 07:52:27 executor [INFO] Processing received message
> source: __system:-1, stream: __tick, id: {}, [5]
>
> Got in the storm ui zeros in emitted and transfered
>
> i executed the command that launch worker and supervisor got
>
> 2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33)
> 2015-12-17 07:59:04 util [ERROR] Async loop died!
>  org.zeromq.ZMQException: Address already in use(0x62) at
> org.zeromq.ZMQ$Socket.bind(Native Method) at zilch.mq$bind.invoke(mq.clj:69)
> at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
> at
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:701)
> 2015-12-17 07:59:04 util [INFO] Halting process:
>
>
> i read that " It was indeed a port conflict, but not with another ZMQ
> process. It turns out our ephemeral port range was messed up on the
> machines:
>
>
> $ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu
> 14.04 i tried it to put 6706 in reserved port but problem still
>
>
> On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
> > Hi Sam,
> >
> > If the worker(pid=2621) belongs to the topology you just submitted, it
> > means the worker has bound the port "6703" successfully.
> > So there should not be any "binding error". Is there any other problems?
> >
> > Regards
> > Basti
> >
> > -----Original Message-----
> > From: sam mohel [mailto:sammohel5@gmail.com]
> > Sent: Thursday, December 17, 2015 2:11 PM
> > To: dev@storm.apache.org
> > Subject: Re: Problem with storm since 4 months
> >
> > can i find help ?
> >
> > On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:
> >
> > > this topology that has problem , i mean this i'm now submitted it
> > >
> > > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu)
> > > <ba...@alibaba-inc.com>
> > > wrote:
> > >
> > >> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> > >> Please check if this topology has already been killed. If so, just
> > >> kill this process.
> > >>
> > >> Regards
> > >> Basti
> > >> -----Original Message-----
> > >> From: sam mohel [mailto:sammohel5@gmail.com]
> > >> Sent: Friday, December 11, 2015 11:18 AM
> > >> To: dev@storm.apache.org
> > >> Subject: Re: Problem with storm since 4 months
> > >>
> > >> is that right command ps aux |grep 2621
> > >> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
> > >> -server -Djava.net.preferIPv4Stack=true
> > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> > >> -Dlog4j.configuration=storm.log.properties -cp
> > >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/c
> > >> om
> > >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/h
> > >> om
> > >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/sto
> > >> rm
> > >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.j
> > >> ar
> > >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0
> > >> .8
> > >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/serv
> > >> le
> > >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5
> > >> .8
> > >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/use
> > >> r/
> > >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/
> > >> lo
> > >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/use
> > >> r/
> > >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet
> > >> -a
> > >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/
> > >> st
> > >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-sim
> > >> pl
> > >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAP
> > >> SH
> > >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/s
> > >> to
> > >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubat
> > >> or
> > >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/st
> > >> or
> > >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3
> > >> .j
> > >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0
> > >> .8
> > >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-
> > >> se
> > >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/
> > >> ho
> > >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jett
> > >> y-
> > >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/h
> > >> om
> > >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib
> > >> /j
> > >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/ho
> > >> me
> > >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/li
> > >> b/
> > >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3
> > >> .3
> > >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHO
> > >> T-
> > >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
> > >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2
> > >> /l
> > >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
> > >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/s
> > >> to
> > >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeya
> > >> ml
> > >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user
> > >> /s
> > >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj
> > >> -t
> > >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/c
> > >> on
> > >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/sto
> > >> rm jar.jar backtype.storm.daemon.worker fsd-1-1449794574
> > >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> > >> ff8603ef-5fb9-47c7-90d0-01813ace4188
> > >> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
> > >> --color=auto 2621
> > >>
> > >>
> > >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com>
> wrote:
> > >>
> > >> > please , how can i check it ?
> > >> >
> > >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
> > >> > <ba...@alibaba-inc.com>
> > >> > wrote:
> > >> >
> > >> >> Okay. Please check which application this process belongs to,
> > >> >> and whether is it OK to kill it.
> > >> >>
> > >> >> Regards
> > >> >> Basti
> > >> >> -----Original Message-----
> > >> >> From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> Sent: Friday, December 11, 2015 11:06 AM
> > >> >> To: dev@storm.apache.org
> > >> >> Subject: Re: Problem with storm since 4 months
> > >> >>
> > >> >> i used
> > >> >>
> > >> >>  netstat -ntape
> > >> >>
> > >> >> and got
> > >> >> Code:
> > >> >>
> > >> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> > >> >> LISTEN      1000       22776       2621/java
> > >> >>
> > >> >> then used this command with PID of PORT = 2621
> > >> >> Code:
> > >> >>
> > >> >>  lsof -P -w -n -p 2621 -a -i
> > >> >>
> > >> >> and got
> > >> >> Code:
> > >> >>
> > >> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> > >> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> > >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> > >> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> > >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> > >> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
> > >> >>
> > >> >>
> > >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
> > >> >> <ba...@alibaba-inc.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Sam,
> > >> >> >
> > >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
> > >> >> > What is the result?
> > >> >> > We need to find which process bound this port, then kill the
> > >> >> > process to release this port.
> > >> >> >
> > >> >> > Regards
> > >> >> > Basti
> > >> >> >
> > >> >> > -----Original Message-----
> > >> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> > Sent: Friday, December 11, 2015 8:15 AM
> > >> >> > To: dev@storm.apache.org
> > >> >> > Subject: Re: Problem with storm since 4 months
> > >> >> >
> > >> >> > > now i converted the change
> > >> >> >
> > >> >> > but problem still
> > >> >> >
> > >> >> > > i mean in the link you sent , he said "the ephermal port
> > >> >> > > range is
> > >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
> > >> >> > probably extend it to run from 16k to 64k."
> > >> >> >
> > >> >> > how can i do that ?
> > >> >> >
> > >> >> > > how can i check if there is anothe dump worker working ,
> > >> >> > or
> > >> >> > is there another way to kill any worker if it is dump or not
> > >> >> "completely"
> > >> >> > and start from scratch
> > >> >> >
> > >> >> > thanks for your patience and your time
> > >> >> >
> > >> >> >
> > >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> > >> >> > eweathers@groupon.com.invalid> wrote:
> > >> >> >
> > >> >> > > responses inline.
> > >> >> > >
> > >> >> > > >Ah, so, you shouldn't have touched the "local port range"
> > >> >> > > >setting since
> > >> >> > > you
> > >> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> > >> >> > >
> > >> >> > > I'll give a brief synopsis:
> > >> >> > >
> > >> >> > > Say you're making a connection from a client to a server
> > >> >> > > that listens on port 80. The client needs to have a port of
> > >> >> > > its own to receive the response packets from the server. The
> > >> >> > > port that is allocated for it by the TCP stack is a "random"
> > >> >> > > port, which is called an "ephemeral" port in
> > >> >> > this context.
> > >> >> > > So with your original default config (32768    61000) the port
> > >> would
> > >> >> be
> > >> >> > > chosen from an available port on the machine that lies
> > >> >> > > within that range of
> > >> >> > > 32768->61000.  That's good, it wouldn't conflict with the
> > >> >> > > 32768->default TCP ports
> > >> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
> > >> >> > > isn't your problem.  But now you've made it possible for it
> > >> >> > > to become a problem, since now the range of ports that can
> > >> >> > > be given as an ephemeral port overlaps with the default
> > >> >> > > Storm Worker ports
> > >> >> > > (1024->65535 includes 67xx).  So you should revert that
> > >> >> > > config
> > >> change.
> > >> >> > >
> > >> >> > > Thanks for your replying , now i reverted the change
> > >> >> >
> > >> >> > >
> > >> >> > > >Sorry, I don't understand what you are asking. What thing
> > >> >> > > >is
> > "16"
> > >> >> > > >that
> > >> >> > > you
> > >> >> > > are trying to extend?
> > >> >> > >
> > >> >> > > i mean in the link you sent , he adviced to extend from 16
> > >> >> > > to
> > >> >> > > 64
> > >> >> > > >You can kill topologies from the Nimbus UI (web page).  Or
> > >> >> > > >with the
> > >> >> > > bin/storm command.
> > >> >> > >
> > >> >> > >
> > >> >> > > >The commands are *solely* intended to figure out what is
> > >> conflicting.
> > >> >> > > Your
> > >> >> > > logs claim that there is something holding onto 67xx which
> > >> >> > > prevents
> > >> >> the
> > >> >> > > Storm Worker from launching.   So if that is happening you
> should
> > >> >> > > *immediately* try to figure out what is actually holding
> > >> >> > > onto the port and preventing your Storm Worker from launching.
> > >> >> > >
> > >> >> > >
> > >> >> > > >Not sure what you mean here.
> > >> >> > >
> > >> >> > > Please note that Storm often suffers from "cascading"
> > >> >> > > failures, where there are a lot of exceptions and errors
> > >> >> > > that aren't actually the root cause of the problem.  Often
> > >> >> > > you need to spend time and effort looking at lots of logs
> > >> >> > > and tracing back to the real root
> > >> >> cause.
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > > >
> > >> >> > > > Really , Thanks for your time
> > >> >> > > >
> > >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > >> >> > > > eweathers@groupon.com.invalid> wrote:
> > >> >> > > >
> > >> >> > > > > Regarding Basti's suggestion (1.) that your host's
> > >> >> > > > > configured ephemeral ports might be conflicting with the
> > >> >> > > > > storm worker ports, here's how you
> > >> >> > > > can
> > >> >> > > > > check your "local port range" setting:
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-
> > >> >> > > te
> > >> >> > > ll- how -m any-ephemeral-ports-are-left-available
> > >> >> > > > >
> > >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > It's possible that there is a zombie worker process
> > >> >> > > > > holding onto port
> > >> >> > > > 6703.
> > >> >> > > > > I would try to identify the process like so:
> > >> >> > > > >
> > >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w
> > >> >> > > > > LISTEN
> > >> >> > > > > | grep -w
> > >> >> > > > 6703
> > >> >> > > > >
> > >> >> > > > > Alternatively you can try a global lsof search:
> > >> >> > > > >
> > >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > >> >> > > > >
> > >> >> > > > > - Erik
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > >> >> > > basti.lj@alibaba-inc.com>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi Sam,
> > >> >> > > > > >
> > >> >> > > > > > You can try to find which process has bound this port
> > >> >> > > > > > by "netstat
> > >> >> > > -anp"
> > >> >> > > > > > first.
> > >> >> > > > > >
> > >> >> > > > > > Generally, there are following cases for the binding
> error.
> > >> >> > > > > > 1. "local port range" is not set to exclude the port
> > >> >> > > > > > range used in
> > >> >> > > > Storm.
> > >> >> > > > > > 2. The previous worker was not killed correctly.
> > >> >> > > > > > 3. There is bug of assignment in some scenarios. Same
> > >> >> > > > > > port was
> > >> >> > > assigned
> > >> >> > > > > to
> > >> >> > > > > > two workers.
> > >> >> > > > > >
> > >> >> > > > > > Regards
> > >> >> > > > > > Basti
> > >> >> > > > > >
> > >> >> > > > > > -----Original Message-----
> > >> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > >> >> > > > > > To: dev@storm.apache.org
> > >> >> > > > > > Subject: Re: Problem with storm since 4 months
> > >> >> > > > > >
> > >> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> > >> >> > > > > >
> > >> >> > > > > > cannot bind port 6703 i think it's same problem
> > >> >> > > > > >
> > >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha
> > >> >> > > > > > <st...@harsha.io>
> > >> wrote:
> > >> >> > > > > >
> > >> >> > > > > > > Sam,
> > >> >> > > > > > >           you might be using very old version of
> > >> >> > > > > > > storm since its
> > >> >> > > > > showing
> > >> >> > > > > > >           ZeroMQ. Can you try using newer version
> > >> >> > > > > > > storm without
> > >> >> > > zero
> > >> >> > > > > mq.
> > >> >> > > > > > > -Harsha
> > >> >> > > > > > >
> > >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > >> >> > > > > > > > I have this problem since 4months when I submitted
> > >> >> > > > > > > > topology I got this in the worker log file [ERROR]
> > >> >> > > > > > > > Async
> > >> >> loop died!
> > >> >> > > > > > org.zeromq.ZMQException:
> > >> >> > > > > > > > Address already in use(0x62) at
> > >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.c
> > >> >> > > > > > > > lj
> > >> >> > > > > > > > :57
> > >> >> > > > > > > > )at
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG
> > >> >> > > > _$
> > >> >> > > > fn_
> > >> >> > > > _16
> > >> >> > > > 29
> > >> >> > > > .i
> > >> >> > > > > > > nvoke(loader.clj:26)
> > >> >> > > > > > > > at
> > >> >> > > > > > > >
> backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> > >> >> > > > > > > > 375
> > >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> > >> >> > > > java.lang.Thread.run(Unknown
> > >> >> > > > > > > > Source)
> > >> >> > > > > > > >
> > >> >> > > > > > > > when i tried to connect port 6703 and 6702
> > >> >> > > > > > > >
> > >> >> > > > > > > > And supervisor log file hadn't still start
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > I searched everywhere but cannot find any solution
> > >> >> > > > > > > > I hope you can
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >
> >
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
Hi Sam,

Not sure what the reason is why ZMQ cause this error.
If you are using a new version Storm(e.g 0.9.x), could you try to use Netty for message transport, since ZMQ is no longer used in Storm now?
Please make sure following configuration is correct. 
- storm.messaging.transport: "backtype.storm.messaging.netty.Context"

Regards
Basti

-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Thursday, December 17, 2015 5:08 PM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

i edited to make topology.debug true and got in the supervisor log file still hasn't start and in the worker log file

2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"]
2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33]
2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33)
2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33]
2015-12-17 07:52:23 worker [INFO] Launching receive-thread for
5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706
2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33)

after finished loading got alot of this lines

2015-12-17 07:52:27 executor [INFO] Processing received message
source: __system:-1, stream: __tick, id: {}, [5]
2015-12-17 07:52:27 executor [INFO] Processing received message
source: __system:-1, stream: __tick, id: {}, [5]

Got in the storm ui zeros in emitted and transfered

i executed the command that launch worker and supervisor got

2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33)
2015-12-17 07:59:04 util [ERROR] Async loop died!
 org.zeromq.ZMQException: Address already in use(0x62) at org.zeromq.ZMQ$Socket.bind(Native Method) at zilch.mq$bind.invoke(mq.clj:69) at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
at  backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:701)
2015-12-17 07:59:04 util [INFO] Halting process:


i read that " It was indeed a port conflict, but not with another ZMQ process. It turns out our ephemeral port range was messed up on the machines:


$ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu
14.04 i tried it to put 6706 in reserved port but problem still


On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> If the worker(pid=2621) belongs to the topology you just submitted, it 
> means the worker has bound the port "6703" successfully.
> So there should not be any "binding error". Is there any other problems?
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Thursday, December 17, 2015 2:11 PM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> can i find help ?
>
> On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:
>
> > this topology that has problem , i mean this i'm now submitted it
> >
> > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) 
> > <ba...@alibaba-inc.com>
> > wrote:
> >
> >> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> >> Please check if this topology has already been killed. If so, just 
> >> kill this process.
> >>
> >> Regards
> >> Basti
> >> -----Original Message-----
> >> From: sam mohel [mailto:sammohel5@gmail.com]
> >> Sent: Friday, December 11, 2015 11:18 AM
> >> To: dev@storm.apache.org
> >> Subject: Re: Problem with storm since 4 months
> >>
> >> is that right command ps aux |grep 2621
> >> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
> >> -server -Djava.net.preferIPv4Stack=true 
> >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> >> -Dlog4j.configuration=storm.log.properties -cp 
> >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/c
> >> om 
> >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/h
> >> om 
> >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/sto
> >> rm 
> >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.j
> >> ar
> >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0
> >> .8 
> >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/serv
> >> le
> >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5
> >> .8 
> >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/use
> >> r/ 
> >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/
> >> lo 
> >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/use
> >> r/ 
> >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet
> >> -a 
> >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/
> >> st 
> >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-sim
> >> pl 
> >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAP
> >> SH 
> >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/s
> >> to 
> >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubat
> >> or 
> >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/st
> >> or 
> >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3
> >> .j
> >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0
> >> .8 
> >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-
> >> se 
> >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/
> >> ho
> >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jett
> >> y- 
> >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/h
> >> om 
> >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib
> >> /j 
> >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/ho
> >> me 
> >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/li
> >> b/
> >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3
> >> .3
> >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHO
> >> T- 
> >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
> >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2
> >> /l 
> >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
> >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/s
> >> to 
> >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeya
> >> ml 
> >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user
> >> /s 
> >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj
> >> -t 
> >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/c
> >> on 
> >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/sto
> >> rm jar.jar backtype.storm.daemon.worker fsd-1-1449794574
> >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> >> ff8603ef-5fb9-47c7-90d0-01813ace4188
> >> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
> >> --color=auto 2621
> >>
> >>
> >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:
> >>
> >> > please , how can i check it ?
> >> >
> >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) 
> >> > <ba...@alibaba-inc.com>
> >> > wrote:
> >> >
> >> >> Okay. Please check which application this process belongs to, 
> >> >> and whether is it OK to kill it.
> >> >>
> >> >> Regards
> >> >> Basti
> >> >> -----Original Message-----
> >> >> From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> Sent: Friday, December 11, 2015 11:06 AM
> >> >> To: dev@storm.apache.org
> >> >> Subject: Re: Problem with storm since 4 months
> >> >>
> >> >> i used
> >> >>
> >> >>  netstat -ntape
> >> >>
> >> >> and got
> >> >> Code:
> >> >>
> >> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> >> >> LISTEN      1000       22776       2621/java
> >> >>
> >> >> then used this command with PID of PORT = 2621
> >> >> Code:
> >> >>
> >> >>  lsof -P -w -n -p 2621 -a -i
> >> >>
> >> >> and got
> >> >> Code:
> >> >>
> >> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> >> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> >> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> >> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
> >> >>
> >> >>
> >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) 
> >> >> <ba...@alibaba-inc.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Sam,
> >> >> >
> >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
> >> >> > What is the result?
> >> >> > We need to find which process bound this port, then kill the 
> >> >> > process to release this port.
> >> >> >
> >> >> > Regards
> >> >> > Basti
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> > Sent: Friday, December 11, 2015 8:15 AM
> >> >> > To: dev@storm.apache.org
> >> >> > Subject: Re: Problem with storm since 4 months
> >> >> >
> >> >> > > now i converted the change
> >> >> >
> >> >> > but problem still
> >> >> >
> >> >> > > i mean in the link you sent , he said "the ephermal port 
> >> >> > > range is
> >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can 
> >> >> > probably extend it to run from 16k to 64k."
> >> >> >
> >> >> > how can i do that ?
> >> >> >
> >> >> > > how can i check if there is anothe dump worker working ,
> >> >> > or
> >> >> > is there another way to kill any worker if it is dump or not
> >> >> "completely"
> >> >> > and start from scratch
> >> >> >
> >> >> > thanks for your patience and your time
> >> >> >
> >> >> >
> >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < 
> >> >> > eweathers@groupon.com.invalid> wrote:
> >> >> >
> >> >> > > responses inline.
> >> >> > >
> >> >> > > >Ah, so, you shouldn't have touched the "local port range"
> >> >> > > >setting since
> >> >> > > you
> >> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> >> >> > >
> >> >> > > I'll give a brief synopsis:
> >> >> > >
> >> >> > > Say you're making a connection from a client to a server 
> >> >> > > that listens on port 80. The client needs to have a port of 
> >> >> > > its own to receive the response packets from the server. The 
> >> >> > > port that is allocated for it by the TCP stack is a "random" 
> >> >> > > port, which is called an "ephemeral" port in
> >> >> > this context.
> >> >> > > So with your original default config (32768    61000) the port
> >> would
> >> >> be
> >> >> > > chosen from an available port on the machine that lies 
> >> >> > > within that range of
> >> >> > > 32768->61000.  That's good, it wouldn't conflict with the 
> >> >> > > 32768->default TCP ports
> >> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
> >> >> > > isn't your problem.  But now you've made it possible for it 
> >> >> > > to become a problem, since now the range of ports that can 
> >> >> > > be given as an ephemeral port overlaps with the default 
> >> >> > > Storm Worker ports
> >> >> > > (1024->65535 includes 67xx).  So you should revert that 
> >> >> > > config
> >> change.
> >> >> > >
> >> >> > > Thanks for your replying , now i reverted the change
> >> >> >
> >> >> > >
> >> >> > > >Sorry, I don't understand what you are asking. What thing 
> >> >> > > >is
> "16"
> >> >> > > >that
> >> >> > > you
> >> >> > > are trying to extend?
> >> >> > >
> >> >> > > i mean in the link you sent , he adviced to extend from 16 
> >> >> > > to
> >> >> > > 64
> >> >> > > >You can kill topologies from the Nimbus UI (web page).  Or 
> >> >> > > >with the
> >> >> > > bin/storm command.
> >> >> > >
> >> >> > >
> >> >> > > >The commands are *solely* intended to figure out what is
> >> conflicting.
> >> >> > > Your
> >> >> > > logs claim that there is something holding onto 67xx which 
> >> >> > > prevents
> >> >> the
> >> >> > > Storm Worker from launching.   So if that is happening you should
> >> >> > > *immediately* try to figure out what is actually holding 
> >> >> > > onto the port and preventing your Storm Worker from launching.
> >> >> > >
> >> >> > >
> >> >> > > >Not sure what you mean here.
> >> >> > >
> >> >> > > Please note that Storm often suffers from "cascading"
> >> >> > > failures, where there are a lot of exceptions and errors 
> >> >> > > that aren't actually the root cause of the problem.  Often 
> >> >> > > you need to spend time and effort looking at lots of logs 
> >> >> > > and tracing back to the real root
> >> >> cause.
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > >
> >> >> > > > Really , Thanks for your time
> >> >> > > >
> >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < 
> >> >> > > > eweathers@groupon.com.invalid> wrote:
> >> >> > > >
> >> >> > > > > Regarding Basti's suggestion (1.) that your host's 
> >> >> > > > > configured ephemeral ports might be conflicting with the 
> >> >> > > > > storm worker ports, here's how you
> >> >> > > > can
> >> >> > > > > check your "local port range" setting:
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-
> >> >> > > te
> >> >> > > ll- how -m any-ephemeral-ports-are-left-available
> >> >> > > > >
> >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > It's possible that there is a zombie worker process 
> >> >> > > > > holding onto port
> >> >> > > > 6703.
> >> >> > > > > I would try to identify the process like so:
> >> >> > > > >
> >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w 
> >> >> > > > > LISTEN
> >> >> > > > > | grep -w
> >> >> > > > 6703
> >> >> > > > >
> >> >> > > > > Alternatively you can try a global lsof search:
> >> >> > > > >
> >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> >> >> > > > >
> >> >> > > > > - Erik
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> >> >> > > basti.lj@alibaba-inc.com>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > Hi Sam,
> >> >> > > > > >
> >> >> > > > > > You can try to find which process has bound this port 
> >> >> > > > > > by "netstat
> >> >> > > -anp"
> >> >> > > > > > first.
> >> >> > > > > >
> >> >> > > > > > Generally, there are following cases for the binding error.
> >> >> > > > > > 1. "local port range" is not set to exclude the port 
> >> >> > > > > > range used in
> >> >> > > > Storm.
> >> >> > > > > > 2. The previous worker was not killed correctly.
> >> >> > > > > > 3. There is bug of assignment in some scenarios. Same 
> >> >> > > > > > port was
> >> >> > > assigned
> >> >> > > > > to
> >> >> > > > > > two workers.
> >> >> > > > > >
> >> >> > > > > > Regards
> >> >> > > > > > Basti
> >> >> > > > > >
> >> >> > > > > > -----Original Message-----
> >> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> >> >> > > > > > To: dev@storm.apache.org
> >> >> > > > > > Subject: Re: Problem with storm since 4 months
> >> >> > > > > >
> >> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> >> >> > > > > >
> >> >> > > > > > cannot bind port 6703 i think it's same problem
> >> >> > > > > >
> >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha 
> >> >> > > > > > <st...@harsha.io>
> >> wrote:
> >> >> > > > > >
> >> >> > > > > > > Sam,
> >> >> > > > > > >           you might be using very old version of 
> >> >> > > > > > > storm since its
> >> >> > > > > showing
> >> >> > > > > > >           ZeroMQ. Can you try using newer version 
> >> >> > > > > > > storm without
> >> >> > > zero
> >> >> > > > > mq.
> >> >> > > > > > > -Harsha
> >> >> > > > > > >
> >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> >> >> > > > > > > > I have this problem since 4months when I submitted 
> >> >> > > > > > > > topology I got this in the worker log file [ERROR] 
> >> >> > > > > > > > Async
> >> >> loop died!
> >> >> > > > > > org.zeromq.ZMQException:
> >> >> > > > > > > > Address already in use(0x62) at 
> >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at 
> >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.c
> >> >> > > > > > > > lj
> >> >> > > > > > > > :57
> >> >> > > > > > > > )at
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG
> >> >> > > > _$
> >> >> > > > fn_
> >> >> > > > _16
> >> >> > > > 29
> >> >> > > > .i
> >> >> > > > > > > nvoke(loader.clj:26)
> >> >> > > > > > > > at
> >> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> >> >> > > > > > > > 375
> >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> >> >> > > > java.lang.Thread.run(Unknown
> >> >> > > > > > > > Source)
> >> >> > > > > > > >
> >> >> > > > > > > > when i tried to connect port 6703 and 6702
> >> >> > > > > > > >
> >> >> > > > > > > > And supervisor log file hadn't still start
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > I searched everywhere but cannot find any solution 
> >> >> > > > > > > > I hope you can
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >>
> >>
> >
>
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
thanks , the defualt i have 32768    61000 so should i change to  1024
65000 or what ? i checked the port
java      17896 user   86u  IPv4 234437      0t0  TCP *:6706 (LISTEN)


On Thu, Dec 17, 2015 at 11:15 AM, Cody Innowhere <e....@gmail.com>
wrote:

> sam,
> try 'lsof -i4 | grep 6706' to see which process is binding this port.
> also, your setting says that any process that binds a random port may bind
> to a port between 1024~65000, if possible, you can change this to
> 10240~65000 so that no random-binding process will use ports less than
> 10240.
>
> On Thu, Dec 17, 2015 at 5:07 PM, sam mohel <sa...@gmail.com> wrote:
>
> > i edited to make topology.debug true and got in the supervisor log file
> > still
> > hasn't start and in the worker log file
> >
> > 2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"]
> > 2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33]
> > 2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33)
> > 2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33]
> > 2015-12-17 07:52:23 worker [INFO] Launching receive-thread for
> > 5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706
> > 2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33)
> >
> > after finished loading got alot of this lines
> >
> > 2015-12-17 07:52:27 executor [INFO] Processing received message
> > source: __system:-1, stream: __tick, id: {}, [5]
> > 2015-12-17 07:52:27 executor [INFO] Processing received message
> > source: __system:-1, stream: __tick, id: {}, [5]
> >
> > Got in the storm ui zeros in emitted and transfered
> >
> > i executed the command that launch worker and supervisor got
> >
> > 2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33)
> > 2015-12-17 07:59:04 util [ERROR] Async loop died!
> >  org.zeromq.ZMQException: Address already in use(0x62)
> > at org.zeromq.ZMQ$Socket.bind(Native Method)
> > at zilch.mq$bind.invoke(mq.clj:69)
> > at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
> > at
> >
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > at clojure.lang.AFn.run(AFn.java:24)
> > at java.lang.Thread.run(Thread.java:701)
> > 2015-12-17 07:59:04 util [INFO] Halting process:
> >
> >
> > i read that " It was indeed a port conflict, but not
> > with another ZMQ process. It turns out our ephemeral port range was
> messed
> > up on the machines:
> >
> >
> > $ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu
> > 14.04 i tried it to put 6706 in reserved port but problem still
> >
> >
> > On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <basti.lj@alibaba-inc.com
> >
> > wrote:
> >
> > > Hi Sam,
> > >
> > > If the worker(pid=2621) belongs to the topology you just submitted, it
> > > means the worker has bound the port "6703" successfully.
> > > So there should not be any "binding error". Is there any other
> problems?
> > >
> > > Regards
> > > Basti
> > >
> > > -----Original Message-----
> > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > Sent: Thursday, December 17, 2015 2:11 PM
> > > To: dev@storm.apache.org
> > > Subject: Re: Problem with storm since 4 months
> > >
> > > can i find help ?
> > >
> > > On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com>
> wrote:
> > >
> > > > this topology that has problem , i mean this i'm now submitted it
> > > >
> > > > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu)
> > > > <ba...@alibaba-inc.com>
> > > > wrote:
> > > >
> > > >> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> > > >> Please check if this topology has already been killed. If so, just
> > > >> kill this process.
> > > >>
> > > >> Regards
> > > >> Basti
> > > >> -----Original Message-----
> > > >> From: sam mohel [mailto:sammohel5@gmail.com]
> > > >> Sent: Friday, December 11, 2015 11:18 AM
> > > >> To: dev@storm.apache.org
> > > >> Subject: Re: Problem with storm since 4 months
> > > >>
> > > >> is that right command ps aux |grep 2621
> > > >> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01
> java
> > > >> -server -Djava.net.preferIPv4Stack=true
> > > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > > >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> > > >> -Dlog4j.configuration=storm.log.properties -cp
> > > >>
> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/com
> > > >>
> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/hom
> > > >>
> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm
> > > >>
> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar
> > > >>
> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8
> > > >>
> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servle
> > > >>
> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8
> > > >>
> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/
> > > >>
> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/lo
> > > >>
> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/
> > > >>
> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-a
> > > >>
> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/st
> > > >>
> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simpl
> > > >>
> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSH
> > > >>
> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/sto
> > > >>
> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator
> > > >>
> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/stor
> > > >>
> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.j
> > > >>
> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8
> > > >>
> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-se
> > > >>
> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ho
> > > >>
> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-
> > > >>
> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/hom
> > > >>
> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/j
> > > >>
> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home
> > > >>
> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/
> > > >>
> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3
> > > >>
> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-
> > > >>
> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
> > > >>
> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/l
> > > >>
> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
> > > >>
> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/sto
> > > >>
> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml
> > > >>
> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/s
> > > >>
> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-t
> > > >>
> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/con
> > > >>
> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/storm
> > > >> jar.jar backtype.storm.daemon.worker fsd-1-1449794574
> > > >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> > > >> ff8603ef-5fb9-47c7-90d0-01813ace4188
> > > >> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00
> grep
> > > >> --color=auto 2621
> > > >>
> > > >>
> > > >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com>
> > wrote:
> > > >>
> > > >> > please , how can i check it ?
> > > >> >
> > > >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
> > > >> > <ba...@alibaba-inc.com>
> > > >> > wrote:
> > > >> >
> > > >> >> Okay. Please check which application this process belongs to, and
> > > >> >> whether is it OK to kill it.
> > > >> >>
> > > >> >> Regards
> > > >> >> Basti
> > > >> >> -----Original Message-----
> > > >> >> From: sam mohel [mailto:sammohel5@gmail.com]
> > > >> >> Sent: Friday, December 11, 2015 11:06 AM
> > > >> >> To: dev@storm.apache.org
> > > >> >> Subject: Re: Problem with storm since 4 months
> > > >> >>
> > > >> >> i used
> > > >> >>
> > > >> >>  netstat -ntape
> > > >> >>
> > > >> >> and got
> > > >> >> Code:
> > > >> >>
> > > >> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> > > >> >> LISTEN      1000       22776       2621/java
> > > >> >>
> > > >> >> then used this command with PID of PORT = 2621
> > > >> >> Code:
> > > >> >>
> > > >> >>  lsof -P -w -n -p 2621 -a -i
> > > >> >>
> > > >> >> and got
> > > >> >> Code:
> > > >> >>
> > > >> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> > > >> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> > > >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> > > >> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> > > >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> > > >> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703
> (LISTEN)
> > > >> >>
> > > >> >>
> > > >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
> > > >> >> <ba...@alibaba-inc.com>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Hi Sam,
> > > >> >> >
> > > >> >> > Have you tried "netstat" and "lsof" commands recommended by
> Erik?
> > > >> >> > What is the result?
> > > >> >> > We need to find which process bound this port, then kill the
> > > >> >> > process to release this port.
> > > >> >> >
> > > >> >> > Regards
> > > >> >> > Basti
> > > >> >> >
> > > >> >> > -----Original Message-----
> > > >> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> > > >> >> > Sent: Friday, December 11, 2015 8:15 AM
> > > >> >> > To: dev@storm.apache.org
> > > >> >> > Subject: Re: Problem with storm since 4 months
> > > >> >> >
> > > >> >> > > now i converted the change
> > > >> >> >
> > > >> >> > but problem still
> > > >> >> >
> > > >> >> > > i mean in the link you sent , he said "the ephermal port
> range
> > > >> >> > > is
> > > >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
> > > >> >> > probably extend it to run from 16k to 64k."
> > > >> >> >
> > > >> >> > how can i do that ?
> > > >> >> >
> > > >> >> > > how can i check if there is anothe dump worker working ,
> > > >> >> > or
> > > >> >> > is there another way to kill any worker if it is dump or not
> > > >> >> "completely"
> > > >> >> > and start from scratch
> > > >> >> >
> > > >> >> > thanks for your patience and your time
> > > >> >> >
> > > >> >> >
> > > >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> > > >> >> > eweathers@groupon.com.invalid> wrote:
> > > >> >> >
> > > >> >> > > responses inline.
> > > >> >> > >
> > > >> >> > > >Ah, so, you shouldn't have touched the "local port range"
> > > >> >> > > >setting since
> > > >> >> > > you
> > > >> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> > > >> >> > >
> > > >> >> > > I'll give a brief synopsis:
> > > >> >> > >
> > > >> >> > > Say you're making a connection from a client to a server that
> > > >> >> > > listens on port 80. The client needs to have a port of its
> own
> > > >> >> > > to receive the response packets from the server. The port
> that
> > > >> >> > > is allocated for it by the TCP stack is a "random" port,
> which
> > > >> >> > > is called an "ephemeral" port in
> > > >> >> > this context.
> > > >> >> > > So with your original default config (32768    61000) the
> port
> > > >> would
> > > >> >> be
> > > >> >> > > chosen from an available port on the machine that lies within
> > > >> >> > > that range of
> > > >> >> > > 32768->61000.  That's good, it wouldn't conflict with the
> > > >> >> > > 32768->default TCP ports
> > > >> >> > > being listened to by the Storm Worker processes (67xx).  So
> 1.
> > > >> >> > > isn't your problem.  But now you've made it possible for it
> to
> > > >> >> > > become a problem, since now the range of ports that can be
> > > >> >> > > given as an ephemeral port overlaps with the default Storm
> > > >> >> > > Worker ports
> > > >> >> > > (1024->65535 includes 67xx).  So you should revert that
> config
> > > >> change.
> > > >> >> > >
> > > >> >> > > Thanks for your replying , now i reverted the change
> > > >> >> >
> > > >> >> > >
> > > >> >> > > >Sorry, I don't understand what you are asking. What thing is
> > > "16"
> > > >> >> > > >that
> > > >> >> > > you
> > > >> >> > > are trying to extend?
> > > >> >> > >
> > > >> >> > > i mean in the link you sent , he adviced to extend from 16 to
> > > >> >> > > 64
> > > >> >> > > >You can kill topologies from the Nimbus UI (web page).  Or
> > > >> >> > > >with the
> > > >> >> > > bin/storm command.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > >The commands are *solely* intended to figure out what is
> > > >> conflicting.
> > > >> >> > > Your
> > > >> >> > > logs claim that there is something holding onto 67xx which
> > > >> >> > > prevents
> > > >> >> the
> > > >> >> > > Storm Worker from launching.   So if that is happening you
> > should
> > > >> >> > > *immediately* try to figure out what is actually holding onto
> > > >> >> > > the port and preventing your Storm Worker from launching.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > >Not sure what you mean here.
> > > >> >> > >
> > > >> >> > > Please note that Storm often suffers from "cascading"
> > > >> >> > > failures, where there are a lot of exceptions and errors that
> > > >> >> > > aren't actually the root cause of the problem.  Often you
> need
> > > >> >> > > to spend time and effort looking at lots of logs and tracing
> > > >> >> > > back to the real root
> > > >> >> cause.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > >
> > > >> >> > > > Really , Thanks for your time
> > > >> >> > > >
> > > >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > > >> >> > > > eweathers@groupon.com.invalid> wrote:
> > > >> >> > > >
> > > >> >> > > > > Regarding Basti's suggestion (1.) that your host's
> > > >> >> > > > > configured ephemeral ports might be conflicting with the
> > > >> >> > > > > storm worker ports, here's how you
> > > >> >> > > > can
> > > >> >> > > > > check your "local port range" setting:
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> https://serverfault.com/questions/261663/on-linux-how-can-i-te
> > > >> >> > > ll- how -m any-ephemeral-ports-are-left-available
> > > >> >> > > > >
> > > >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > It's possible that there is a zombie worker process
> > > >> >> > > > > holding onto port
> > > >> >> > > > 6703.
> > > >> >> > > > > I would try to identify the process like so:
> > > >> >> > > > >
> > > >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w
> > > >> >> > > > > LISTEN
> > > >> >> > > > > | grep -w
> > > >> >> > > > 6703
> > > >> >> > > > >
> > > >> >> > > > > Alternatively you can try a global lsof search:
> > > >> >> > > > >
> > > >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > > >> >> > > > >
> > > >> >> > > > > - Erik
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > >
> > > >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > > >> >> > > basti.lj@alibaba-inc.com>
> > > >> >> > > > > wrote:
> > > >> >> > > > >
> > > >> >> > > > > > Hi Sam,
> > > >> >> > > > > >
> > > >> >> > > > > > You can try to find which process has bound this port
> by
> > > >> >> > > > > > "netstat
> > > >> >> > > -anp"
> > > >> >> > > > > > first.
> > > >> >> > > > > >
> > > >> >> > > > > > Generally, there are following cases for the binding
> > error.
> > > >> >> > > > > > 1. "local port range" is not set to exclude the port
> > > >> >> > > > > > range used in
> > > >> >> > > > Storm.
> > > >> >> > > > > > 2. The previous worker was not killed correctly.
> > > >> >> > > > > > 3. There is bug of assignment in some scenarios. Same
> > > >> >> > > > > > port was
> > > >> >> > > assigned
> > > >> >> > > > > to
> > > >> >> > > > > > two workers.
> > > >> >> > > > > >
> > > >> >> > > > > > Regards
> > > >> >> > > > > > Basti
> > > >> >> > > > > >
> > > >> >> > > > > > -----Original Message-----
> > > >> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > >> >> > > > > > To: dev@storm.apache.org
> > > >> >> > > > > > Subject: Re: Problem with storm since 4 months
> > > >> >> > > > > >
> > > >> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> > > >> >> > > > > >
> > > >> >> > > > > > cannot bind port 6703 i think it's same problem
> > > >> >> > > > > >
> > > >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <
> storm@harsha.io>
> > > >> wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > > Sam,
> > > >> >> > > > > > >           you might be using very old version of
> storm
> > > >> >> > > > > > > since its
> > > >> >> > > > > showing
> > > >> >> > > > > > >           ZeroMQ. Can you try using newer version
> > > >> >> > > > > > > storm without
> > > >> >> > > zero
> > > >> >> > > > > mq.
> > > >> >> > > > > > > -Harsha
> > > >> >> > > > > > >
> > > >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > >> >> > > > > > > > I have this problem since 4months when I submitted
> > > >> >> > > > > > > > topology I got this in the worker log file [ERROR]
> > > >> >> > > > > > > > Async
> > > >> >> loop died!
> > > >> >> > > > > > org.zeromq.ZMQException:
> > > >> >> > > > > > > > Address already in use(0x62) at
> > > >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > >> >> > > > > > > >
> backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj
> > > >> >> > > > > > > > :57
> > > >> >> > > > > > > > )at
> > > >> >> > > > > > > >
> > > >> >> > > > > > >
> > > >> >> > > >
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$
> > > >> >> > > > fn_
> > > >> >> > > > _16
> > > >> >> > > > 29
> > > >> >> > > > .i
> > > >> >> > > > > > > nvoke(loader.clj:26)
> > > >> >> > > > > > > > at
> > > >> >> > > > > > > >
> > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> > > >> >> > > > > > > > 375
> > > >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> > > >> >> > > > java.lang.Thread.run(Unknown
> > > >> >> > > > > > > > Source)
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > when i tried to connect port 6703 and 6702
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > And supervisor log file hadn't still start
> > > >> >> > > > > > > >
> > > >> >> > > > > > > >
> > > >> >> > > > > > > > I searched everywhere but cannot find any solution
> I
> > > >> >> > > > > > > > hope you can
> > > >> >> > > > > > >
> > > >> >> > > > > >
> > > >> >> > > > > >
> > > >> >> > > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>

Re: Problem with storm since 4 months

Posted by Cody Innowhere <e....@gmail.com>.
sam,
try 'lsof -i4 | grep 6706' to see which process is binding this port.
also, your setting says that any process that binds a random port may bind
to a port between 1024~65000, if possible, you can change this to
10240~65000 so that no random-binding process will use ports less than
10240.

On Thu, Dec 17, 2015 at 5:07 PM, sam mohel <sa...@gmail.com> wrote:

> i edited to make topology.debug true and got in the supervisor log file
> still
> hasn't start and in the worker log file
>
> 2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"]
> 2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33]
> 2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33)
> 2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33]
> 2015-12-17 07:52:23 worker [INFO] Launching receive-thread for
> 5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706
> 2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33)
>
> after finished loading got alot of this lines
>
> 2015-12-17 07:52:27 executor [INFO] Processing received message
> source: __system:-1, stream: __tick, id: {}, [5]
> 2015-12-17 07:52:27 executor [INFO] Processing received message
> source: __system:-1, stream: __tick, id: {}, [5]
>
> Got in the storm ui zeros in emitted and transfered
>
> i executed the command that launch worker and supervisor got
>
> 2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33)
> 2015-12-17 07:59:04 util [ERROR] Async loop died!
>  org.zeromq.ZMQException: Address already in use(0x62)
> at org.zeromq.ZMQ$Socket.bind(Native Method)
> at zilch.mq$bind.invoke(mq.clj:69)
> at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
> at
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> at clojure.lang.AFn.run(AFn.java:24)
> at java.lang.Thread.run(Thread.java:701)
> 2015-12-17 07:59:04 util [INFO] Halting process:
>
>
> i read that " It was indeed a port conflict, but not
> with another ZMQ process. It turns out our ephemeral port range was messed
> up on the machines:
>
>
> $ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu
> 14.04 i tried it to put 6706 in reserved port but problem still
>
>
> On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
> > Hi Sam,
> >
> > If the worker(pid=2621) belongs to the topology you just submitted, it
> > means the worker has bound the port "6703" successfully.
> > So there should not be any "binding error". Is there any other problems?
> >
> > Regards
> > Basti
> >
> > -----Original Message-----
> > From: sam mohel [mailto:sammohel5@gmail.com]
> > Sent: Thursday, December 17, 2015 2:11 PM
> > To: dev@storm.apache.org
> > Subject: Re: Problem with storm since 4 months
> >
> > can i find help ?
> >
> > On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:
> >
> > > this topology that has problem , i mean this i'm now submitted it
> > >
> > > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu)
> > > <ba...@alibaba-inc.com>
> > > wrote:
> > >
> > >> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> > >> Please check if this topology has already been killed. If so, just
> > >> kill this process.
> > >>
> > >> Regards
> > >> Basti
> > >> -----Original Message-----
> > >> From: sam mohel [mailto:sammohel5@gmail.com]
> > >> Sent: Friday, December 11, 2015 11:18 AM
> > >> To: dev@storm.apache.org
> > >> Subject: Re: Problem with storm since 4 months
> > >>
> > >> is that right command ps aux |grep 2621
> > >> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
> > >> -server -Djava.net.preferIPv4Stack=true
> > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> > >> -Dlog4j.configuration=storm.log.properties -cp
> > >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/com
> > >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/hom
> > >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm
> > >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar
> > >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8
> > >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servle
> > >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8
> > >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/
> > >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/lo
> > >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/
> > >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-a
> > >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/st
> > >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simpl
> > >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSH
> > >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/sto
> > >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator
> > >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/stor
> > >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.j
> > >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8
> > >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-se
> > >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ho
> > >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-
> > >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/hom
> > >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/j
> > >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home
> > >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/
> > >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3
> > >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-
> > >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
> > >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/l
> > >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
> > >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/sto
> > >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml
> > >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/s
> > >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-t
> > >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/con
> > >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/storm
> > >> jar.jar backtype.storm.daemon.worker fsd-1-1449794574
> > >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> > >> ff8603ef-5fb9-47c7-90d0-01813ace4188
> > >> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
> > >> --color=auto 2621
> > >>
> > >>
> > >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com>
> wrote:
> > >>
> > >> > please , how can i check it ?
> > >> >
> > >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
> > >> > <ba...@alibaba-inc.com>
> > >> > wrote:
> > >> >
> > >> >> Okay. Please check which application this process belongs to, and
> > >> >> whether is it OK to kill it.
> > >> >>
> > >> >> Regards
> > >> >> Basti
> > >> >> -----Original Message-----
> > >> >> From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> Sent: Friday, December 11, 2015 11:06 AM
> > >> >> To: dev@storm.apache.org
> > >> >> Subject: Re: Problem with storm since 4 months
> > >> >>
> > >> >> i used
> > >> >>
> > >> >>  netstat -ntape
> > >> >>
> > >> >> and got
> > >> >> Code:
> > >> >>
> > >> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> > >> >> LISTEN      1000       22776       2621/java
> > >> >>
> > >> >> then used this command with PID of PORT = 2621
> > >> >> Code:
> > >> >>
> > >> >>  lsof -P -w -n -p 2621 -a -i
> > >> >>
> > >> >> and got
> > >> >> Code:
> > >> >>
> > >> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> > >> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> > >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> > >> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> > >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> > >> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
> > >> >>
> > >> >>
> > >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
> > >> >> <ba...@alibaba-inc.com>
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Sam,
> > >> >> >
> > >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
> > >> >> > What is the result?
> > >> >> > We need to find which process bound this port, then kill the
> > >> >> > process to release this port.
> > >> >> >
> > >> >> > Regards
> > >> >> > Basti
> > >> >> >
> > >> >> > -----Original Message-----
> > >> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> > Sent: Friday, December 11, 2015 8:15 AM
> > >> >> > To: dev@storm.apache.org
> > >> >> > Subject: Re: Problem with storm since 4 months
> > >> >> >
> > >> >> > > now i converted the change
> > >> >> >
> > >> >> > but problem still
> > >> >> >
> > >> >> > > i mean in the link you sent , he said "the ephermal port range
> > >> >> > > is
> > >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
> > >> >> > probably extend it to run from 16k to 64k."
> > >> >> >
> > >> >> > how can i do that ?
> > >> >> >
> > >> >> > > how can i check if there is anothe dump worker working ,
> > >> >> > or
> > >> >> > is there another way to kill any worker if it is dump or not
> > >> >> "completely"
> > >> >> > and start from scratch
> > >> >> >
> > >> >> > thanks for your patience and your time
> > >> >> >
> > >> >> >
> > >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> > >> >> > eweathers@groupon.com.invalid> wrote:
> > >> >> >
> > >> >> > > responses inline.
> > >> >> > >
> > >> >> > > >Ah, so, you shouldn't have touched the "local port range"
> > >> >> > > >setting since
> > >> >> > > you
> > >> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> > >> >> > >
> > >> >> > > I'll give a brief synopsis:
> > >> >> > >
> > >> >> > > Say you're making a connection from a client to a server that
> > >> >> > > listens on port 80. The client needs to have a port of its own
> > >> >> > > to receive the response packets from the server. The port that
> > >> >> > > is allocated for it by the TCP stack is a "random" port, which
> > >> >> > > is called an "ephemeral" port in
> > >> >> > this context.
> > >> >> > > So with your original default config (32768    61000) the port
> > >> would
> > >> >> be
> > >> >> > > chosen from an available port on the machine that lies within
> > >> >> > > that range of
> > >> >> > > 32768->61000.  That's good, it wouldn't conflict with the
> > >> >> > > 32768->default TCP ports
> > >> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
> > >> >> > > isn't your problem.  But now you've made it possible for it to
> > >> >> > > become a problem, since now the range of ports that can be
> > >> >> > > given as an ephemeral port overlaps with the default Storm
> > >> >> > > Worker ports
> > >> >> > > (1024->65535 includes 67xx).  So you should revert that config
> > >> change.
> > >> >> > >
> > >> >> > > Thanks for your replying , now i reverted the change
> > >> >> >
> > >> >> > >
> > >> >> > > >Sorry, I don't understand what you are asking. What thing is
> > "16"
> > >> >> > > >that
> > >> >> > > you
> > >> >> > > are trying to extend?
> > >> >> > >
> > >> >> > > i mean in the link you sent , he adviced to extend from 16 to
> > >> >> > > 64
> > >> >> > > >You can kill topologies from the Nimbus UI (web page).  Or
> > >> >> > > >with the
> > >> >> > > bin/storm command.
> > >> >> > >
> > >> >> > >
> > >> >> > > >The commands are *solely* intended to figure out what is
> > >> conflicting.
> > >> >> > > Your
> > >> >> > > logs claim that there is something holding onto 67xx which
> > >> >> > > prevents
> > >> >> the
> > >> >> > > Storm Worker from launching.   So if that is happening you
> should
> > >> >> > > *immediately* try to figure out what is actually holding onto
> > >> >> > > the port and preventing your Storm Worker from launching.
> > >> >> > >
> > >> >> > >
> > >> >> > > >Not sure what you mean here.
> > >> >> > >
> > >> >> > > Please note that Storm often suffers from "cascading"
> > >> >> > > failures, where there are a lot of exceptions and errors that
> > >> >> > > aren't actually the root cause of the problem.  Often you need
> > >> >> > > to spend time and effort looking at lots of logs and tracing
> > >> >> > > back to the real root
> > >> >> cause.
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > >
> > >> >> > > >
> > >> >> > > > Really , Thanks for your time
> > >> >> > > >
> > >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > >> >> > > > eweathers@groupon.com.invalid> wrote:
> > >> >> > > >
> > >> >> > > > > Regarding Basti's suggestion (1.) that your host's
> > >> >> > > > > configured ephemeral ports might be conflicting with the
> > >> >> > > > > storm worker ports, here's how you
> > >> >> > > > can
> > >> >> > > > > check your "local port range" setting:
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-te
> > >> >> > > ll- how -m any-ephemeral-ports-are-left-available
> > >> >> > > > >
> > >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > It's possible that there is a zombie worker process
> > >> >> > > > > holding onto port
> > >> >> > > > 6703.
> > >> >> > > > > I would try to identify the process like so:
> > >> >> > > > >
> > >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w
> > >> >> > > > > LISTEN
> > >> >> > > > > | grep -w
> > >> >> > > > 6703
> > >> >> > > > >
> > >> >> > > > > Alternatively you can try a global lsof search:
> > >> >> > > > >
> > >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > >> >> > > > >
> > >> >> > > > > - Erik
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > >
> > >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > >> >> > > basti.lj@alibaba-inc.com>
> > >> >> > > > > wrote:
> > >> >> > > > >
> > >> >> > > > > > Hi Sam,
> > >> >> > > > > >
> > >> >> > > > > > You can try to find which process has bound this port by
> > >> >> > > > > > "netstat
> > >> >> > > -anp"
> > >> >> > > > > > first.
> > >> >> > > > > >
> > >> >> > > > > > Generally, there are following cases for the binding
> error.
> > >> >> > > > > > 1. "local port range" is not set to exclude the port
> > >> >> > > > > > range used in
> > >> >> > > > Storm.
> > >> >> > > > > > 2. The previous worker was not killed correctly.
> > >> >> > > > > > 3. There is bug of assignment in some scenarios. Same
> > >> >> > > > > > port was
> > >> >> > > assigned
> > >> >> > > > > to
> > >> >> > > > > > two workers.
> > >> >> > > > > >
> > >> >> > > > > > Regards
> > >> >> > > > > > Basti
> > >> >> > > > > >
> > >> >> > > > > > -----Original Message-----
> > >> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > >> >> > > > > > To: dev@storm.apache.org
> > >> >> > > > > > Subject: Re: Problem with storm since 4 months
> > >> >> > > > > >
> > >> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> > >> >> > > > > >
> > >> >> > > > > > cannot bind port 6703 i think it's same problem
> > >> >> > > > > >
> > >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io>
> > >> wrote:
> > >> >> > > > > >
> > >> >> > > > > > > Sam,
> > >> >> > > > > > >           you might be using very old version of storm
> > >> >> > > > > > > since its
> > >> >> > > > > showing
> > >> >> > > > > > >           ZeroMQ. Can you try using newer version
> > >> >> > > > > > > storm without
> > >> >> > > zero
> > >> >> > > > > mq.
> > >> >> > > > > > > -Harsha
> > >> >> > > > > > >
> > >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > >> >> > > > > > > > I have this problem since 4months when I submitted
> > >> >> > > > > > > > topology I got this in the worker log file [ERROR]
> > >> >> > > > > > > > Async
> > >> >> loop died!
> > >> >> > > > > > org.zeromq.ZMQException:
> > >> >> > > > > > > > Address already in use(0x62) at
> > >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj
> > >> >> > > > > > > > :57
> > >> >> > > > > > > > )at
> > >> >> > > > > > > >
> > >> >> > > > > > >
> > >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$
> > >> >> > > > fn_
> > >> >> > > > _16
> > >> >> > > > 29
> > >> >> > > > .i
> > >> >> > > > > > > nvoke(loader.clj:26)
> > >> >> > > > > > > > at
> > >> >> > > > > > > >
> backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> > >> >> > > > > > > > 375
> > >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> > >> >> > > > java.lang.Thread.run(Unknown
> > >> >> > > > > > > > Source)
> > >> >> > > > > > > >
> > >> >> > > > > > > > when i tried to connect port 6703 and 6702
> > >> >> > > > > > > >
> > >> >> > > > > > > > And supervisor log file hadn't still start
> > >> >> > > > > > > >
> > >> >> > > > > > > >
> > >> >> > > > > > > > I searched everywhere but cannot find any solution I
> > >> >> > > > > > > > hope you can
> > >> >> > > > > > >
> > >> >> > > > > >
> > >> >> > > > > >
> > >> >> > > > >
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >
> >
> >
>

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
i edited to make topology.debug true and got in the supervisor log file still
hasn't start and in the worker log file

2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"]
2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33]
2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33)
2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33]
2015-12-17 07:52:23 worker [INFO] Launching receive-thread for
5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706
2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33)

after finished loading got alot of this lines

2015-12-17 07:52:27 executor [INFO] Processing received message
source: __system:-1, stream: __tick, id: {}, [5]
2015-12-17 07:52:27 executor [INFO] Processing received message
source: __system:-1, stream: __tick, id: {}, [5]

Got in the storm ui zeros in emitted and transfered

i executed the command that launch worker and supervisor got

2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33)
2015-12-17 07:59:04 util [ERROR] Async loop died!
 org.zeromq.ZMQException: Address already in use(0x62)
at org.zeromq.ZMQ$Socket.bind(Native Method)
at zilch.mq$bind.invoke(mq.clj:69)
at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)
at  backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:701)
2015-12-17 07:59:04 util [INFO] Halting process:


i read that " It was indeed a port conflict, but not
with another ZMQ process. It turns out our ephemeral port range was messed
up on the machines:


$ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu
14.04 i tried it to put 6706 in reserved port but problem still


On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> If the worker(pid=2621) belongs to the topology you just submitted, it
> means the worker has bound the port "6703" successfully.
> So there should not be any "binding error". Is there any other problems?
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Thursday, December 17, 2015 2:11 PM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> can i find help ?
>
> On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:
>
> > this topology that has problem , i mean this i'm now submitted it
> >
> > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu)
> > <ba...@alibaba-inc.com>
> > wrote:
> >
> >> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> >> Please check if this topology has already been killed. If so, just
> >> kill this process.
> >>
> >> Regards
> >> Basti
> >> -----Original Message-----
> >> From: sam mohel [mailto:sammohel5@gmail.com]
> >> Sent: Friday, December 11, 2015 11:18 AM
> >> To: dev@storm.apache.org
> >> Subject: Re: Problem with storm since 4 months
> >>
> >> is that right command ps aux |grep 2621
> >> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
> >> -server -Djava.net.preferIPv4Stack=true
> >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> >> -Dlog4j.configuration=storm.log.properties -cp
> >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/com
> >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/hom
> >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm
> >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar
> >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8
> >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servle
> >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8
> >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/
> >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/lo
> >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/
> >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-a
> >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/st
> >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simpl
> >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSH
> >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/sto
> >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator
> >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/stor
> >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.j
> >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8
> >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-se
> >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ho
> >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-
> >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/hom
> >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/j
> >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home
> >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/
> >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3
> >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-
> >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
> >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/l
> >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
> >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/sto
> >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml
> >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/s
> >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-t
> >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/con
> >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/storm
> >> jar.jar backtype.storm.daemon.worker fsd-1-1449794574
> >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> >> ff8603ef-5fb9-47c7-90d0-01813ace4188
> >> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
> >> --color=auto 2621
> >>
> >>
> >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:
> >>
> >> > please , how can i check it ?
> >> >
> >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
> >> > <ba...@alibaba-inc.com>
> >> > wrote:
> >> >
> >> >> Okay. Please check which application this process belongs to, and
> >> >> whether is it OK to kill it.
> >> >>
> >> >> Regards
> >> >> Basti
> >> >> -----Original Message-----
> >> >> From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> Sent: Friday, December 11, 2015 11:06 AM
> >> >> To: dev@storm.apache.org
> >> >> Subject: Re: Problem with storm since 4 months
> >> >>
> >> >> i used
> >> >>
> >> >>  netstat -ntape
> >> >>
> >> >> and got
> >> >> Code:
> >> >>
> >> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> >> >> LISTEN      1000       22776       2621/java
> >> >>
> >> >> then used this command with PID of PORT = 2621
> >> >> Code:
> >> >>
> >> >>  lsof -P -w -n -p 2621 -a -i
> >> >>
> >> >> and got
> >> >> Code:
> >> >>
> >> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> >> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> >> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> >> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
> >> >>
> >> >>
> >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
> >> >> <ba...@alibaba-inc.com>
> >> >> wrote:
> >> >>
> >> >> > Hi Sam,
> >> >> >
> >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
> >> >> > What is the result?
> >> >> > We need to find which process bound this port, then kill the
> >> >> > process to release this port.
> >> >> >
> >> >> > Regards
> >> >> > Basti
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> > Sent: Friday, December 11, 2015 8:15 AM
> >> >> > To: dev@storm.apache.org
> >> >> > Subject: Re: Problem with storm since 4 months
> >> >> >
> >> >> > > now i converted the change
> >> >> >
> >> >> > but problem still
> >> >> >
> >> >> > > i mean in the link you sent , he said "the ephermal port range
> >> >> > > is
> >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
> >> >> > probably extend it to run from 16k to 64k."
> >> >> >
> >> >> > how can i do that ?
> >> >> >
> >> >> > > how can i check if there is anothe dump worker working ,
> >> >> > or
> >> >> > is there another way to kill any worker if it is dump or not
> >> >> "completely"
> >> >> > and start from scratch
> >> >> >
> >> >> > thanks for your patience and your time
> >> >> >
> >> >> >
> >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> >> >> > eweathers@groupon.com.invalid> wrote:
> >> >> >
> >> >> > > responses inline.
> >> >> > >
> >> >> > > >Ah, so, you shouldn't have touched the "local port range"
> >> >> > > >setting since
> >> >> > > you
> >> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> >> >> > >
> >> >> > > I'll give a brief synopsis:
> >> >> > >
> >> >> > > Say you're making a connection from a client to a server that
> >> >> > > listens on port 80. The client needs to have a port of its own
> >> >> > > to receive the response packets from the server. The port that
> >> >> > > is allocated for it by the TCP stack is a "random" port, which
> >> >> > > is called an "ephemeral" port in
> >> >> > this context.
> >> >> > > So with your original default config (32768    61000) the port
> >> would
> >> >> be
> >> >> > > chosen from an available port on the machine that lies within
> >> >> > > that range of
> >> >> > > 32768->61000.  That's good, it wouldn't conflict with the
> >> >> > > 32768->default TCP ports
> >> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
> >> >> > > isn't your problem.  But now you've made it possible for it to
> >> >> > > become a problem, since now the range of ports that can be
> >> >> > > given as an ephemeral port overlaps with the default Storm
> >> >> > > Worker ports
> >> >> > > (1024->65535 includes 67xx).  So you should revert that config
> >> change.
> >> >> > >
> >> >> > > Thanks for your replying , now i reverted the change
> >> >> >
> >> >> > >
> >> >> > > >Sorry, I don't understand what you are asking. What thing is
> "16"
> >> >> > > >that
> >> >> > > you
> >> >> > > are trying to extend?
> >> >> > >
> >> >> > > i mean in the link you sent , he adviced to extend from 16 to
> >> >> > > 64
> >> >> > > >You can kill topologies from the Nimbus UI (web page).  Or
> >> >> > > >with the
> >> >> > > bin/storm command.
> >> >> > >
> >> >> > >
> >> >> > > >The commands are *solely* intended to figure out what is
> >> conflicting.
> >> >> > > Your
> >> >> > > logs claim that there is something holding onto 67xx which
> >> >> > > prevents
> >> >> the
> >> >> > > Storm Worker from launching.   So if that is happening you should
> >> >> > > *immediately* try to figure out what is actually holding onto
> >> >> > > the port and preventing your Storm Worker from launching.
> >> >> > >
> >> >> > >
> >> >> > > >Not sure what you mean here.
> >> >> > >
> >> >> > > Please note that Storm often suffers from "cascading"
> >> >> > > failures, where there are a lot of exceptions and errors that
> >> >> > > aren't actually the root cause of the problem.  Often you need
> >> >> > > to spend time and effort looking at lots of logs and tracing
> >> >> > > back to the real root
> >> >> cause.
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > >
> >> >> > > > Really , Thanks for your time
> >> >> > > >
> >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> >> >> > > > eweathers@groupon.com.invalid> wrote:
> >> >> > > >
> >> >> > > > > Regarding Basti's suggestion (1.) that your host's
> >> >> > > > > configured ephemeral ports might be conflicting with the
> >> >> > > > > storm worker ports, here's how you
> >> >> > > > can
> >> >> > > > > check your "local port range" setting:
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-te
> >> >> > > ll- how -m any-ephemeral-ports-are-left-available
> >> >> > > > >
> >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > It's possible that there is a zombie worker process
> >> >> > > > > holding onto port
> >> >> > > > 6703.
> >> >> > > > > I would try to identify the process like so:
> >> >> > > > >
> >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w
> >> >> > > > > LISTEN
> >> >> > > > > | grep -w
> >> >> > > > 6703
> >> >> > > > >
> >> >> > > > > Alternatively you can try a global lsof search:
> >> >> > > > >
> >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> >> >> > > > >
> >> >> > > > > - Erik
> >> >> > > > >
> >> >> > > > >
> >> >> > > > >
> >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> >> >> > > basti.lj@alibaba-inc.com>
> >> >> > > > > wrote:
> >> >> > > > >
> >> >> > > > > > Hi Sam,
> >> >> > > > > >
> >> >> > > > > > You can try to find which process has bound this port by
> >> >> > > > > > "netstat
> >> >> > > -anp"
> >> >> > > > > > first.
> >> >> > > > > >
> >> >> > > > > > Generally, there are following cases for the binding error.
> >> >> > > > > > 1. "local port range" is not set to exclude the port
> >> >> > > > > > range used in
> >> >> > > > Storm.
> >> >> > > > > > 2. The previous worker was not killed correctly.
> >> >> > > > > > 3. There is bug of assignment in some scenarios. Same
> >> >> > > > > > port was
> >> >> > > assigned
> >> >> > > > > to
> >> >> > > > > > two workers.
> >> >> > > > > >
> >> >> > > > > > Regards
> >> >> > > > > > Basti
> >> >> > > > > >
> >> >> > > > > > -----Original Message-----
> >> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> >> >> > > > > > To: dev@storm.apache.org
> >> >> > > > > > Subject: Re: Problem with storm since 4 months
> >> >> > > > > >
> >> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> >> >> > > > > >
> >> >> > > > > > cannot bind port 6703 i think it's same problem
> >> >> > > > > >
> >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io>
> >> wrote:
> >> >> > > > > >
> >> >> > > > > > > Sam,
> >> >> > > > > > >           you might be using very old version of storm
> >> >> > > > > > > since its
> >> >> > > > > showing
> >> >> > > > > > >           ZeroMQ. Can you try using newer version
> >> >> > > > > > > storm without
> >> >> > > zero
> >> >> > > > > mq.
> >> >> > > > > > > -Harsha
> >> >> > > > > > >
> >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> >> >> > > > > > > > I have this problem since 4months when I submitted
> >> >> > > > > > > > topology I got this in the worker log file [ERROR]
> >> >> > > > > > > > Async
> >> >> loop died!
> >> >> > > > > > org.zeromq.ZMQException:
> >> >> > > > > > > > Address already in use(0x62) at
> >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj
> >> >> > > > > > > > :57
> >> >> > > > > > > > )at
> >> >> > > > > > > >
> >> >> > > > > > >
> >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$
> >> >> > > > fn_
> >> >> > > > _16
> >> >> > > > 29
> >> >> > > > .i
> >> >> > > > > > > nvoke(loader.clj:26)
> >> >> > > > > > > > at
> >> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> >> >> > > > > > > > 375
> >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> >> >> > > > java.lang.Thread.run(Unknown
> >> >> > > > > > > > Source)
> >> >> > > > > > > >
> >> >> > > > > > > > when i tried to connect port 6703 and 6702
> >> >> > > > > > > >
> >> >> > > > > > > > And supervisor log file hadn't still start
> >> >> > > > > > > >
> >> >> > > > > > > >
> >> >> > > > > > > > I searched everywhere but cannot find any solution I
> >> >> > > > > > > > hope you can
> >> >> > > > > > >
> >> >> > > > > >
> >> >> > > > > >
> >> >> > > > >
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >>
> >>
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
Hi Sam,

If the worker(pid=2621) belongs to the topology you just submitted, it means the worker has bound the port "6703" successfully.
So there should not be any "binding error". Is there any other problems?

Regards
Basti

-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Thursday, December 17, 2015 2:11 PM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

can i find help ?

On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:

> this topology that has problem , i mean this i'm now submitted it
>
> On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) 
> <ba...@alibaba-inc.com>
> wrote:
>
>> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
>> Please check if this topology has already been killed. If so, just 
>> kill this process.
>>
>> Regards
>> Basti
>> -----Original Message-----
>> From: sam mohel [mailto:sammohel5@gmail.com]
>> Sent: Friday, December 11, 2015 11:18 AM
>> To: dev@storm.apache.org
>> Subject: Re: Problem with storm since 4 months
>>
>> is that right command ps aux |grep 2621
>> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
>> -server -Djava.net.preferIPv4Stack=true 
>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
>> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
>> -Dlog4j.configuration=storm.log.properties -cp 
>> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/com
>> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/hom
>> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm
>> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar
>> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8
>> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servle
>> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8
>> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/
>> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/lo
>> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/
>> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-a
>> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/st
>> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simpl
>> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSH
>> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/sto
>> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator
>> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/stor
>> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.j
>> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8
>> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-se
>> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ho
>> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-
>> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/hom
>> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/j
>> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home
>> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/
>> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3
>> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-
>> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.
>> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/l
>> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.
>> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/sto
>> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml
>> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/s
>> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-t
>> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/con
>> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/storm
>> jar.jar backtype.storm.daemon.worker fsd-1-1449794574
>> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
>> ff8603ef-5fb9-47c7-90d0-01813ace4188
>> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
>> --color=auto 2621
>>
>>
>> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:
>>
>> > please , how can i check it ?
>> >
>> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) 
>> > <ba...@alibaba-inc.com>
>> > wrote:
>> >
>> >> Okay. Please check which application this process belongs to, and 
>> >> whether is it OK to kill it.
>> >>
>> >> Regards
>> >> Basti
>> >> -----Original Message-----
>> >> From: sam mohel [mailto:sammohel5@gmail.com]
>> >> Sent: Friday, December 11, 2015 11:06 AM
>> >> To: dev@storm.apache.org
>> >> Subject: Re: Problem with storm since 4 months
>> >>
>> >> i used
>> >>
>> >>  netstat -ntape
>> >>
>> >> and got
>> >> Code:
>> >>
>> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
>> >> LISTEN      1000       22776       2621/java
>> >>
>> >> then used this command with PID of PORT = 2621
>> >> Code:
>> >>
>> >>  lsof -P -w -n -p 2621 -a -i
>> >>
>> >> and got
>> >> Code:
>> >>
>> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>> >> java    2621 user   76u  IPv4  18906      0t0  TCP
>> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
>> >> java    2621 user   83u  IPv4  22773      0t0  TCP
>> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
>> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
>> >>
>> >>
>> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) 
>> >> <ba...@alibaba-inc.com>
>> >> wrote:
>> >>
>> >> > Hi Sam,
>> >> >
>> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
>> >> > What is the result?
>> >> > We need to find which process bound this port, then kill the 
>> >> > process to release this port.
>> >> >
>> >> > Regards
>> >> > Basti
>> >> >
>> >> > -----Original Message-----
>> >> > From: sam mohel [mailto:sammohel5@gmail.com]
>> >> > Sent: Friday, December 11, 2015 8:15 AM
>> >> > To: dev@storm.apache.org
>> >> > Subject: Re: Problem with storm since 4 months
>> >> >
>> >> > > now i converted the change
>> >> >
>> >> > but problem still
>> >> >
>> >> > > i mean in the link you sent , he said "the ephermal port range 
>> >> > > is
>> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can 
>> >> > probably extend it to run from 16k to 64k."
>> >> >
>> >> > how can i do that ?
>> >> >
>> >> > > how can i check if there is anothe dump worker working ,
>> >> > or
>> >> > is there another way to kill any worker if it is dump or not
>> >> "completely"
>> >> > and start from scratch
>> >> >
>> >> > thanks for your patience and your time
>> >> >
>> >> >
>> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < 
>> >> > eweathers@groupon.com.invalid> wrote:
>> >> >
>> >> > > responses inline.
>> >> > >
>> >> > > >Ah, so, you shouldn't have touched the "local port range"
>> >> > > >setting since
>> >> > > you
>> >> > > don't fully understand ephemeral ports and TCP yet! :-)
>> >> > >
>> >> > > I'll give a brief synopsis:
>> >> > >
>> >> > > Say you're making a connection from a client to a server that 
>> >> > > listens on port 80. The client needs to have a port of its own 
>> >> > > to receive the response packets from the server. The port that 
>> >> > > is allocated for it by the TCP stack is a "random" port, which 
>> >> > > is called an "ephemeral" port in
>> >> > this context.
>> >> > > So with your original default config (32768    61000) the port
>> would
>> >> be
>> >> > > chosen from an available port on the machine that lies within 
>> >> > > that range of
>> >> > > 32768->61000.  That's good, it wouldn't conflict with the 
>> >> > > 32768->default TCP ports
>> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
>> >> > > isn't your problem.  But now you've made it possible for it to 
>> >> > > become a problem, since now the range of ports that can be 
>> >> > > given as an ephemeral port overlaps with the default Storm 
>> >> > > Worker ports
>> >> > > (1024->65535 includes 67xx).  So you should revert that config
>> change.
>> >> > >
>> >> > > Thanks for your replying , now i reverted the change
>> >> >
>> >> > >
>> >> > > >Sorry, I don't understand what you are asking. What thing is "16"
>> >> > > >that
>> >> > > you
>> >> > > are trying to extend?
>> >> > >
>> >> > > i mean in the link you sent , he adviced to extend from 16 to 
>> >> > > 64
>> >> > > >You can kill topologies from the Nimbus UI (web page).  Or 
>> >> > > >with the
>> >> > > bin/storm command.
>> >> > >
>> >> > >
>> >> > > >The commands are *solely* intended to figure out what is
>> conflicting.
>> >> > > Your
>> >> > > logs claim that there is something holding onto 67xx which 
>> >> > > prevents
>> >> the
>> >> > > Storm Worker from launching.   So if that is happening you should
>> >> > > *immediately* try to figure out what is actually holding onto 
>> >> > > the port and preventing your Storm Worker from launching.
>> >> > >
>> >> > >
>> >> > > >Not sure what you mean here.
>> >> > >
>> >> > > Please note that Storm often suffers from "cascading" 
>> >> > > failures, where there are a lot of exceptions and errors that 
>> >> > > aren't actually the root cause of the problem.  Often you need 
>> >> > > to spend time and effort looking at lots of logs and tracing 
>> >> > > back to the real root
>> >> cause.
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > >
>> >> > > > Really , Thanks for your time
>> >> > > >
>> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < 
>> >> > > > eweathers@groupon.com.invalid> wrote:
>> >> > > >
>> >> > > > > Regarding Basti's suggestion (1.) that your host's 
>> >> > > > > configured ephemeral ports might be conflicting with the 
>> >> > > > > storm worker ports, here's how you
>> >> > > > can
>> >> > > > > check your "local port range" setting:
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-te
>> >> > > ll- how -m any-ephemeral-ports-are-left-available
>> >> > > > >
>> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
>> >> > > > >
>> >> > > > >
>> >> > > > > It's possible that there is a zombie worker process 
>> >> > > > > holding onto port
>> >> > > > 6703.
>> >> > > > > I would try to identify the process like so:
>> >> > > > >
>> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w 
>> >> > > > > LISTEN
>> >> > > > > | grep -w
>> >> > > > 6703
>> >> > > > >
>> >> > > > > Alternatively you can try a global lsof search:
>> >> > > > >
>> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
>> >> > > > >
>> >> > > > > - Erik
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
>> >> > > basti.lj@alibaba-inc.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi Sam,
>> >> > > > > >
>> >> > > > > > You can try to find which process has bound this port by 
>> >> > > > > > "netstat
>> >> > > -anp"
>> >> > > > > > first.
>> >> > > > > >
>> >> > > > > > Generally, there are following cases for the binding error.
>> >> > > > > > 1. "local port range" is not set to exclude the port 
>> >> > > > > > range used in
>> >> > > > Storm.
>> >> > > > > > 2. The previous worker was not killed correctly.
>> >> > > > > > 3. There is bug of assignment in some scenarios. Same 
>> >> > > > > > port was
>> >> > > assigned
>> >> > > > > to
>> >> > > > > > two workers.
>> >> > > > > >
>> >> > > > > > Regards
>> >> > > > > > Basti
>> >> > > > > >
>> >> > > > > > -----Original Message-----
>> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
>> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
>> >> > > > > > To: dev@storm.apache.org
>> >> > > > > > Subject: Re: Problem with storm since 4 months
>> >> > > > > >
>> >> > > > > > i tried to use storm-0.9.5 but problem changed with
>> >> > > > > >
>> >> > > > > > cannot bind port 6703 i think it's same problem
>> >> > > > > >
>> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io>
>> wrote:
>> >> > > > > >
>> >> > > > > > > Sam,
>> >> > > > > > >           you might be using very old version of storm 
>> >> > > > > > > since its
>> >> > > > > showing
>> >> > > > > > >           ZeroMQ. Can you try using newer version 
>> >> > > > > > > storm without
>> >> > > zero
>> >> > > > > mq.
>> >> > > > > > > -Harsha
>> >> > > > > > >
>> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
>> >> > > > > > > > I have this problem since 4months when I submitted 
>> >> > > > > > > > topology I got this in the worker log file [ERROR] 
>> >> > > > > > > > Async
>> >> loop died!
>> >> > > > > > org.zeromq.ZMQException:
>> >> > > > > > > > Address already in use(0x62) at 
>> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
>> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
>> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj
>> >> > > > > > > > :57
>> >> > > > > > > > )at
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$
>> >> > > > fn_
>> >> > > > _16
>> >> > > > 29
>> >> > > > .i
>> >> > > > > > > nvoke(loader.clj:26)
>> >> > > > > > > > at
>> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
>> >> > > > > > > > 375
>> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
>> >> > > > java.lang.Thread.run(Unknown
>> >> > > > > > > > Source)
>> >> > > > > > > >
>> >> > > > > > > > when i tried to connect port 6703 and 6702
>> >> > > > > > > >
>> >> > > > > > > > And supervisor log file hadn't still start
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > I searched everywhere but cannot find any solution I 
>> >> > > > > > > > hope you can
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >>
>> >>
>> >
>>
>>
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
can i find help ?

On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <sa...@gmail.com> wrote:

> this topology that has problem , i mean this i'm now submitted it
>
> On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
>> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
>> Please check if this topology has already been killed. If so, just kill
>> this process.
>>
>> Regards
>> Basti
>> -----Original Message-----
>> From: sam mohel [mailto:sammohel5@gmail.com]
>> Sent: Friday, December 11, 2015 11:18 AM
>> To: dev@storm.apache.org
>> Subject: Re: Problem with storm since 4 months
>>
>> is that right command ps aux |grep 2621
>> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
>> -server -Djava.net.preferIPv4Stack=true
>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
>> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
>> -Dlog4j.configuration=storm.log.properties -cp
>> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/commons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm-0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/log4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/storm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simple-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/storm-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/jline-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/lib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/storm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/conf:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/stormjar.jar
>> backtype.storm.daemon.worker fsd-1-1449794574
>> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
>> ff8603ef-5fb9-47c7-90d0-01813ace4188
>> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
>> --color=auto 2621
>>
>>
>> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:
>>
>> > please , how can i check it ?
>> >
>> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
>> > <ba...@alibaba-inc.com>
>> > wrote:
>> >
>> >> Okay. Please check which application this process belongs to, and
>> >> whether is it OK to kill it.
>> >>
>> >> Regards
>> >> Basti
>> >> -----Original Message-----
>> >> From: sam mohel [mailto:sammohel5@gmail.com]
>> >> Sent: Friday, December 11, 2015 11:06 AM
>> >> To: dev@storm.apache.org
>> >> Subject: Re: Problem with storm since 4 months
>> >>
>> >> i used
>> >>
>> >>  netstat -ntape
>> >>
>> >> and got
>> >> Code:
>> >>
>> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
>> >> LISTEN      1000       22776       2621/java
>> >>
>> >> then used this command with PID of PORT = 2621
>> >> Code:
>> >>
>> >>  lsof -P -w -n -p 2621 -a -i
>> >>
>> >> and got
>> >> Code:
>> >>
>> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>> >> java    2621 user   76u  IPv4  18906      0t0  TCP
>> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
>> >> java    2621 user   83u  IPv4  22773      0t0  TCP
>> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
>> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
>> >>
>> >>
>> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
>> >> <ba...@alibaba-inc.com>
>> >> wrote:
>> >>
>> >> > Hi Sam,
>> >> >
>> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
>> >> > What is the result?
>> >> > We need to find which process bound this port, then kill the
>> >> > process to release this port.
>> >> >
>> >> > Regards
>> >> > Basti
>> >> >
>> >> > -----Original Message-----
>> >> > From: sam mohel [mailto:sammohel5@gmail.com]
>> >> > Sent: Friday, December 11, 2015 8:15 AM
>> >> > To: dev@storm.apache.org
>> >> > Subject: Re: Problem with storm since 4 months
>> >> >
>> >> > > now i converted the change
>> >> >
>> >> > but problem still
>> >> >
>> >> > > i mean in the link you sent , he said "the ephermal port range is
>> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
>> >> > probably extend it to run from 16k to 64k."
>> >> >
>> >> > how can i do that ?
>> >> >
>> >> > > how can i check if there is anothe dump worker working ,
>> >> > or
>> >> > is there another way to kill any worker if it is dump or not
>> >> "completely"
>> >> > and start from scratch
>> >> >
>> >> > thanks for your patience and your time
>> >> >
>> >> >
>> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
>> >> > eweathers@groupon.com.invalid> wrote:
>> >> >
>> >> > > responses inline.
>> >> > >
>> >> > > >Ah, so, you shouldn't have touched the "local port range"
>> >> > > >setting since
>> >> > > you
>> >> > > don't fully understand ephemeral ports and TCP yet! :-)
>> >> > >
>> >> > > I'll give a brief synopsis:
>> >> > >
>> >> > > Say you're making a connection from a client to a server that
>> >> > > listens on port 80. The client needs to have a port of its own to
>> >> > > receive the response packets from the server. The port that is
>> >> > > allocated for it by the TCP stack is a "random" port, which is
>> >> > > called an "ephemeral" port in
>> >> > this context.
>> >> > > So with your original default config (32768    61000) the port
>> would
>> >> be
>> >> > > chosen from an available port on the machine that lies within
>> >> > > that range of
>> >> > > 32768->61000.  That's good, it wouldn't conflict with the default
>> >> > > 32768->TCP ports
>> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
>> >> > > isn't your problem.  But now you've made it possible for it to
>> >> > > become a problem, since now the range of ports that can be given
>> >> > > as an ephemeral port overlaps with the default Storm Worker ports
>> >> > > (1024->65535 includes 67xx).  So you should revert that config
>> change.
>> >> > >
>> >> > > Thanks for your replying , now i reverted the change
>> >> >
>> >> > >
>> >> > > >Sorry, I don't understand what you are asking. What thing is "16"
>> >> > > >that
>> >> > > you
>> >> > > are trying to extend?
>> >> > >
>> >> > > i mean in the link you sent , he adviced to extend from 16 to 64
>> >> > > >You can kill topologies from the Nimbus UI (web page).  Or with
>> >> > > >the
>> >> > > bin/storm command.
>> >> > >
>> >> > >
>> >> > > >The commands are *solely* intended to figure out what is
>> conflicting.
>> >> > > Your
>> >> > > logs claim that there is something holding onto 67xx which
>> >> > > prevents
>> >> the
>> >> > > Storm Worker from launching.   So if that is happening you should
>> >> > > *immediately* try to figure out what is actually holding onto the
>> >> > > port and preventing your Storm Worker from launching.
>> >> > >
>> >> > >
>> >> > > >Not sure what you mean here.
>> >> > >
>> >> > > Please note that Storm often suffers from "cascading" failures,
>> >> > > where there are a lot of exceptions and errors that aren't
>> >> > > actually the root cause of the problem.  Often you need to spend
>> >> > > time and effort looking at lots of logs and tracing back to the
>> >> > > real root
>> >> cause.
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > >
>> >> > > > Really , Thanks for your time
>> >> > > >
>> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
>> >> > > > eweathers@groupon.com.invalid> wrote:
>> >> > > >
>> >> > > > > Regarding Basti's suggestion (1.) that your host's configured
>> >> > > > > ephemeral ports might be conflicting with the storm worker
>> >> > > > > ports, here's how you
>> >> > > > can
>> >> > > > > check your "local port range" setting:
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-
>> >> > > how -m any-ephemeral-ports-are-left-available
>> >> > > > >
>> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
>> >> > > > >
>> >> > > > >
>> >> > > > > It's possible that there is a zombie worker process holding
>> >> > > > > onto port
>> >> > > > 6703.
>> >> > > > > I would try to identify the process like so:
>> >> > > > >
>> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN
>> >> > > > > | grep -w
>> >> > > > 6703
>> >> > > > >
>> >> > > > > Alternatively you can try a global lsof search:
>> >> > > > >
>> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
>> >> > > > >
>> >> > > > > - Erik
>> >> > > > >
>> >> > > > >
>> >> > > > >
>> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
>> >> > > basti.lj@alibaba-inc.com>
>> >> > > > > wrote:
>> >> > > > >
>> >> > > > > > Hi Sam,
>> >> > > > > >
>> >> > > > > > You can try to find which process has bound this port by
>> >> > > > > > "netstat
>> >> > > -anp"
>> >> > > > > > first.
>> >> > > > > >
>> >> > > > > > Generally, there are following cases for the binding error.
>> >> > > > > > 1. "local port range" is not set to exclude the port range
>> >> > > > > > used in
>> >> > > > Storm.
>> >> > > > > > 2. The previous worker was not killed correctly.
>> >> > > > > > 3. There is bug of assignment in some scenarios. Same port
>> >> > > > > > was
>> >> > > assigned
>> >> > > > > to
>> >> > > > > > two workers.
>> >> > > > > >
>> >> > > > > > Regards
>> >> > > > > > Basti
>> >> > > > > >
>> >> > > > > > -----Original Message-----
>> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
>> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
>> >> > > > > > To: dev@storm.apache.org
>> >> > > > > > Subject: Re: Problem with storm since 4 months
>> >> > > > > >
>> >> > > > > > i tried to use storm-0.9.5 but problem changed with
>> >> > > > > >
>> >> > > > > > cannot bind port 6703 i think it's same problem
>> >> > > > > >
>> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io>
>> wrote:
>> >> > > > > >
>> >> > > > > > > Sam,
>> >> > > > > > >           you might be using very old version of storm
>> >> > > > > > > since its
>> >> > > > > showing
>> >> > > > > > >           ZeroMQ. Can you try using newer version storm
>> >> > > > > > > without
>> >> > > zero
>> >> > > > > mq.
>> >> > > > > > > -Harsha
>> >> > > > > > >
>> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
>> >> > > > > > > > I have this problem since 4months when I submitted
>> >> > > > > > > > topology I got this in the worker log file [ERROR]
>> >> > > > > > > > Async
>> >> loop died!
>> >> > > > > > org.zeromq.ZMQException:
>> >> > > > > > > > Address already in use(0x62) at
>> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
>> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
>> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57
>> >> > > > > > > > )at
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn_
>> >> > > > _16
>> >> > > > 29
>> >> > > > .i
>> >> > > > > > > nvoke(loader.clj:26)
>> >> > > > > > > > at
>> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
>> >> > > > > > > > 375
>> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
>> >> > > > java.lang.Thread.run(Unknown
>> >> > > > > > > > Source)
>> >> > > > > > > >
>> >> > > > > > > > when i tried to connect port 6703 and 6702
>> >> > > > > > > >
>> >> > > > > > > > And supervisor log file hadn't still start
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > I searched everywhere but cannot find any solution I
>> >> > > > > > > > hope you can
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >>
>> >>
>> >
>>
>>
>

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
this topology that has problem , i mean this i'm now submitted it

On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> This worker (pid=2621) belongs to topology " fsd-1-1449794574".
> Please check if this topology has already been killed. If so, just kill
> this process.
>
> Regards
> Basti
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Friday, December 11, 2015 11:18 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> is that right command ps aux |grep 2621
> user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
> -server -Djava.net.preferIPv4Stack=true
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
> -Dlog4j.configuration=storm.log.properties -cp
> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/commons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm-0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/log4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/storm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simple-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/storm-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/jline-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/lib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/storm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/conf:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/stormjar.jar
> backtype.storm.daemon.worker fsd-1-1449794574
> 1953edf9-8f3e-4bd8-870a-05370272b097 6703
> ff8603ef-5fb9-47c7-90d0-01813ace4188
> user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
> --color=auto 2621
>
>
> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:
>
> > please , how can i check it ?
> >
> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu)
> > <ba...@alibaba-inc.com>
> > wrote:
> >
> >> Okay. Please check which application this process belongs to, and
> >> whether is it OK to kill it.
> >>
> >> Regards
> >> Basti
> >> -----Original Message-----
> >> From: sam mohel [mailto:sammohel5@gmail.com]
> >> Sent: Friday, December 11, 2015 11:06 AM
> >> To: dev@storm.apache.org
> >> Subject: Re: Problem with storm since 4 months
> >>
> >> i used
> >>
> >>  netstat -ntape
> >>
> >> and got
> >> Code:
> >>
> >> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> >> LISTEN      1000       22776       2621/java
> >>
> >> then used this command with PID of PORT = 2621
> >> Code:
> >>
> >>  lsof -P -w -n -p 2621 -a -i
> >>
> >> and got
> >> Code:
> >>
> >> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> >> java    2621 user   76u  IPv4  18906      0t0  TCP
> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> >> java    2621 user   83u  IPv4  22773      0t0  TCP
> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> >> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
> >>
> >>
> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu)
> >> <ba...@alibaba-inc.com>
> >> wrote:
> >>
> >> > Hi Sam,
> >> >
> >> > Have you tried "netstat" and "lsof" commands recommended by Erik?
> >> > What is the result?
> >> > We need to find which process bound this port, then kill the
> >> > process to release this port.
> >> >
> >> > Regards
> >> > Basti
> >> >
> >> > -----Original Message-----
> >> > From: sam mohel [mailto:sammohel5@gmail.com]
> >> > Sent: Friday, December 11, 2015 8:15 AM
> >> > To: dev@storm.apache.org
> >> > Subject: Re: Problem with storm since 4 months
> >> >
> >> > > now i converted the change
> >> >
> >> > but problem still
> >> >
> >> > > i mean in the link you sent , he said "the ephermal port range is
> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can
> >> > probably extend it to run from 16k to 64k."
> >> >
> >> > how can i do that ?
> >> >
> >> > > how can i check if there is anothe dump worker working ,
> >> > or
> >> > is there another way to kill any worker if it is dump or not
> >> "completely"
> >> > and start from scratch
> >> >
> >> > thanks for your patience and your time
> >> >
> >> >
> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> >> > eweathers@groupon.com.invalid> wrote:
> >> >
> >> > > responses inline.
> >> > >
> >> > > >Ah, so, you shouldn't have touched the "local port range"
> >> > > >setting since
> >> > > you
> >> > > don't fully understand ephemeral ports and TCP yet! :-)
> >> > >
> >> > > I'll give a brief synopsis:
> >> > >
> >> > > Say you're making a connection from a client to a server that
> >> > > listens on port 80. The client needs to have a port of its own to
> >> > > receive the response packets from the server. The port that is
> >> > > allocated for it by the TCP stack is a "random" port, which is
> >> > > called an "ephemeral" port in
> >> > this context.
> >> > > So with your original default config (32768    61000) the port would
> >> be
> >> > > chosen from an available port on the machine that lies within
> >> > > that range of
> >> > > 32768->61000.  That's good, it wouldn't conflict with the default
> >> > > 32768->TCP ports
> >> > > being listened to by the Storm Worker processes (67xx).  So 1.
> >> > > isn't your problem.  But now you've made it possible for it to
> >> > > become a problem, since now the range of ports that can be given
> >> > > as an ephemeral port overlaps with the default Storm Worker ports
> >> > > (1024->65535 includes 67xx).  So you should revert that config
> change.
> >> > >
> >> > > Thanks for your replying , now i reverted the change
> >> >
> >> > >
> >> > > >Sorry, I don't understand what you are asking. What thing is "16"
> >> > > >that
> >> > > you
> >> > > are trying to extend?
> >> > >
> >> > > i mean in the link you sent , he adviced to extend from 16 to 64
> >> > > >You can kill topologies from the Nimbus UI (web page).  Or with
> >> > > >the
> >> > > bin/storm command.
> >> > >
> >> > >
> >> > > >The commands are *solely* intended to figure out what is
> conflicting.
> >> > > Your
> >> > > logs claim that there is something holding onto 67xx which
> >> > > prevents
> >> the
> >> > > Storm Worker from launching.   So if that is happening you should
> >> > > *immediately* try to figure out what is actually holding onto the
> >> > > port and preventing your Storm Worker from launching.
> >> > >
> >> > >
> >> > > >Not sure what you mean here.
> >> > >
> >> > > Please note that Storm often suffers from "cascading" failures,
> >> > > where there are a lot of exceptions and errors that aren't
> >> > > actually the root cause of the problem.  Often you need to spend
> >> > > time and effort looking at lots of logs and tracing back to the
> >> > > real root
> >> cause.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > > > Really , Thanks for your time
> >> > > >
> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> >> > > > eweathers@groupon.com.invalid> wrote:
> >> > > >
> >> > > > > Regarding Basti's suggestion (1.) that your host's configured
> >> > > > > ephemeral ports might be conflicting with the storm worker
> >> > > > > ports, here's how you
> >> > > > can
> >> > > > > check your "local port range" setting:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-
> >> > > how -m any-ephemeral-ports-are-left-available
> >> > > > >
> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> >> > > > >
> >> > > > >
> >> > > > > It's possible that there is a zombie worker process holding
> >> > > > > onto port
> >> > > > 6703.
> >> > > > > I would try to identify the process like so:
> >> > > > >
> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN
> >> > > > > | grep -w
> >> > > > 6703
> >> > > > >
> >> > > > > Alternatively you can try a global lsof search:
> >> > > > >
> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> >> > > > >
> >> > > > > - Erik
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> >> > > basti.lj@alibaba-inc.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi Sam,
> >> > > > > >
> >> > > > > > You can try to find which process has bound this port by
> >> > > > > > "netstat
> >> > > -anp"
> >> > > > > > first.
> >> > > > > >
> >> > > > > > Generally, there are following cases for the binding error.
> >> > > > > > 1. "local port range" is not set to exclude the port range
> >> > > > > > used in
> >> > > > Storm.
> >> > > > > > 2. The previous worker was not killed correctly.
> >> > > > > > 3. There is bug of assignment in some scenarios. Same port
> >> > > > > > was
> >> > > assigned
> >> > > > > to
> >> > > > > > two workers.
> >> > > > > >
> >> > > > > > Regards
> >> > > > > > Basti
> >> > > > > >
> >> > > > > > -----Original Message-----
> >> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> >> > > > > > To: dev@storm.apache.org
> >> > > > > > Subject: Re: Problem with storm since 4 months
> >> > > > > >
> >> > > > > > i tried to use storm-0.9.5 but problem changed with
> >> > > > > >
> >> > > > > > cannot bind port 6703 i think it's same problem
> >> > > > > >
> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io>
> wrote:
> >> > > > > >
> >> > > > > > > Sam,
> >> > > > > > >           you might be using very old version of storm
> >> > > > > > > since its
> >> > > > > showing
> >> > > > > > >           ZeroMQ. Can you try using newer version storm
> >> > > > > > > without
> >> > > zero
> >> > > > > mq.
> >> > > > > > > -Harsha
> >> > > > > > >
> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> >> > > > > > > > I have this problem since 4months when I submitted
> >> > > > > > > > topology I got this in the worker log file [ERROR]
> >> > > > > > > > Async
> >> loop died!
> >> > > > > > org.zeromq.ZMQException:
> >> > > > > > > > Address already in use(0x62) at
> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57
> >> > > > > > > > )at
> >> > > > > > > >
> >> > > > > > >
> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn_
> >> > > > _16
> >> > > > 29
> >> > > > .i
> >> > > > > > > nvoke(loader.clj:26)
> >> > > > > > > > at
> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
> >> > > > > > > > 375
> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> >> > > > java.lang.Thread.run(Unknown
> >> > > > > > > > Source)
> >> > > > > > > >
> >> > > > > > > > when i tried to connect port 6703 and 6702
> >> > > > > > > >
> >> > > > > > > > And supervisor log file hadn't still start
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I searched everywhere but cannot find any solution I
> >> > > > > > > > hope you can
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
This worker (pid=2621) belongs to topology " fsd-1-1449794574". 
Please check if this topology has already been killed. If so, just kill this process.

Regards
Basti
-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Friday, December 11, 2015 11:18 AM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

is that right command ps aux |grep 2621
user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
-server -Djava.net.preferIPv4Stack=true
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
-Dlog4j.configuration=storm.log.properties -cp /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/commons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm-0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/log4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/storm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simple-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/storm-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/jline-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/lib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/storm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/conf:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/stormjar.jar
backtype.storm.daemon.worker fsd-1-1449794574
1953edf9-8f3e-4bd8-870a-05370272b097 6703
ff8603ef-5fb9-47c7-90d0-01813ace4188
user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
--color=auto 2621


On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:

> please , how can i check it ?
>
> On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) 
> <ba...@alibaba-inc.com>
> wrote:
>
>> Okay. Please check which application this process belongs to, and 
>> whether is it OK to kill it.
>>
>> Regards
>> Basti
>> -----Original Message-----
>> From: sam mohel [mailto:sammohel5@gmail.com]
>> Sent: Friday, December 11, 2015 11:06 AM
>> To: dev@storm.apache.org
>> Subject: Re: Problem with storm since 4 months
>>
>> i used
>>
>>  netstat -ntape
>>
>> and got
>> Code:
>>
>> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
>> LISTEN      1000       22776       2621/java
>>
>> then used this command with PID of PORT = 2621
>> Code:
>>
>>  lsof -P -w -n -p 2621 -a -i
>>
>> and got
>> Code:
>>
>> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>> java    2621 user   76u  IPv4  18906      0t0  TCP
>> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
>> java    2621 user   83u  IPv4  22773      0t0  TCP
>> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
>> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
>>
>>
>> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) 
>> <ba...@alibaba-inc.com>
>> wrote:
>>
>> > Hi Sam,
>> >
>> > Have you tried "netstat" and "lsof" commands recommended by Erik? 
>> > What is the result?
>> > We need to find which process bound this port, then kill the 
>> > process to release this port.
>> >
>> > Regards
>> > Basti
>> >
>> > -----Original Message-----
>> > From: sam mohel [mailto:sammohel5@gmail.com]
>> > Sent: Friday, December 11, 2015 8:15 AM
>> > To: dev@storm.apache.org
>> > Subject: Re: Problem with storm since 4 months
>> >
>> > > now i converted the change
>> >
>> > but problem still
>> >
>> > > i mean in the link you sent , he said "the ephermal port range is
>> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can 
>> > probably extend it to run from 16k to 64k."
>> >
>> > how can i do that ?
>> >
>> > > how can i check if there is anothe dump worker working ,
>> > or
>> > is there another way to kill any worker if it is dump or not
>> "completely"
>> > and start from scratch
>> >
>> > thanks for your patience and your time
>> >
>> >
>> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < 
>> > eweathers@groupon.com.invalid> wrote:
>> >
>> > > responses inline.
>> > >
>> > > >Ah, so, you shouldn't have touched the "local port range" 
>> > > >setting since
>> > > you
>> > > don't fully understand ephemeral ports and TCP yet! :-)
>> > >
>> > > I'll give a brief synopsis:
>> > >
>> > > Say you're making a connection from a client to a server that 
>> > > listens on port 80. The client needs to have a port of its own to 
>> > > receive the response packets from the server. The port that is 
>> > > allocated for it by the TCP stack is a "random" port, which is 
>> > > called an "ephemeral" port in
>> > this context.
>> > > So with your original default config (32768    61000) the port would
>> be
>> > > chosen from an available port on the machine that lies within 
>> > > that range of
>> > > 32768->61000.  That's good, it wouldn't conflict with the default 
>> > > 32768->TCP ports
>> > > being listened to by the Storm Worker processes (67xx).  So 1. 
>> > > isn't your problem.  But now you've made it possible for it to 
>> > > become a problem, since now the range of ports that can be given 
>> > > as an ephemeral port overlaps with the default Storm Worker ports
>> > > (1024->65535 includes 67xx).  So you should revert that config change.
>> > >
>> > > Thanks for your replying , now i reverted the change
>> >
>> > >
>> > > >Sorry, I don't understand what you are asking. What thing is "16"
>> > > >that
>> > > you
>> > > are trying to extend?
>> > >
>> > > i mean in the link you sent , he adviced to extend from 16 to 64
>> > > >You can kill topologies from the Nimbus UI (web page).  Or with 
>> > > >the
>> > > bin/storm command.
>> > >
>> > >
>> > > >The commands are *solely* intended to figure out what is conflicting.
>> > > Your
>> > > logs claim that there is something holding onto 67xx which 
>> > > prevents
>> the
>> > > Storm Worker from launching.   So if that is happening you should
>> > > *immediately* try to figure out what is actually holding onto the 
>> > > port and preventing your Storm Worker from launching.
>> > >
>> > >
>> > > >Not sure what you mean here.
>> > >
>> > > Please note that Storm often suffers from "cascading" failures, 
>> > > where there are a lot of exceptions and errors that aren't 
>> > > actually the root cause of the problem.  Often you need to spend 
>> > > time and effort looking at lots of logs and tracing back to the 
>> > > real root
>> cause.
>> > >
>> > >
>> > >
>> > >
>> > > >
>> > > > Really , Thanks for your time
>> > > >
>> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < 
>> > > > eweathers@groupon.com.invalid> wrote:
>> > > >
>> > > > > Regarding Basti's suggestion (1.) that your host's configured 
>> > > > > ephemeral ports might be conflicting with the storm worker 
>> > > > > ports, here's how you
>> > > > can
>> > > > > check your "local port range" setting:
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-
>> > > how -m any-ephemeral-ports-are-left-available
>> > > > >
>> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
>> > > > >
>> > > > >
>> > > > > It's possible that there is a zombie worker process holding 
>> > > > > onto port
>> > > > 6703.
>> > > > > I would try to identify the process like so:
>> > > > >
>> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN 
>> > > > > | grep -w
>> > > > 6703
>> > > > >
>> > > > > Alternatively you can try a global lsof search:
>> > > > >
>> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
>> > > > >
>> > > > > - Erik
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
>> > > basti.lj@alibaba-inc.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Sam,
>> > > > > >
>> > > > > > You can try to find which process has bound this port by 
>> > > > > > "netstat
>> > > -anp"
>> > > > > > first.
>> > > > > >
>> > > > > > Generally, there are following cases for the binding error.
>> > > > > > 1. "local port range" is not set to exclude the port range 
>> > > > > > used in
>> > > > Storm.
>> > > > > > 2. The previous worker was not killed correctly.
>> > > > > > 3. There is bug of assignment in some scenarios. Same port 
>> > > > > > was
>> > > assigned
>> > > > > to
>> > > > > > two workers.
>> > > > > >
>> > > > > > Regards
>> > > > > > Basti
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
>> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
>> > > > > > To: dev@storm.apache.org
>> > > > > > Subject: Re: Problem with storm since 4 months
>> > > > > >
>> > > > > > i tried to use storm-0.9.5 but problem changed with
>> > > > > >
>> > > > > > cannot bind port 6703 i think it's same problem
>> > > > > >
>> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
>> > > > > >
>> > > > > > > Sam,
>> > > > > > >           you might be using very old version of storm 
>> > > > > > > since its
>> > > > > showing
>> > > > > > >           ZeroMQ. Can you try using newer version storm 
>> > > > > > > without
>> > > zero
>> > > > > mq.
>> > > > > > > -Harsha
>> > > > > > >
>> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
>> > > > > > > > I have this problem since 4months when I submitted 
>> > > > > > > > topology I got this in the worker log file [ERROR] 
>> > > > > > > > Async
>> loop died!
>> > > > > > org.zeromq.ZMQException:
>> > > > > > > > Address already in use(0x62) at 
>> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
>> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at 
>> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57
>> > > > > > > > )at
>> > > > > > > >
>> > > > > > >
>> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn_
>> > > > _16
>> > > > 29
>> > > > .i
>> > > > > > > nvoke(loader.clj:26)
>> > > > > > > > at
>> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:
>> > > > > > > > 375
>> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
>> > > > java.lang.Thread.run(Unknown
>> > > > > > > > Source)
>> > > > > > > >
>> > > > > > > > when i tried to connect port 6703 and 6702
>> > > > > > > >
>> > > > > > > > And supervisor log file hadn't still start
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I searched everywhere but cannot find any solution I 
>> > > > > > > > hope you can
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>>
>>
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
is that right command ps aux |grep 2621
user      2621  7.8  2.7 3444276 108056 pts/12 Sl+  02:42  12:01 java
-server -Djava.net.preferIPv4Stack=true
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2
-Dlog4j.configuration=storm.log.properties -cp
/home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/commons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/home/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm-0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar:/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8.2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8.jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/log4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-api-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/storm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simple-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/storm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator-0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/storm-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.jar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8.2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-servlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/home/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty-6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/home/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/jline-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home/user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3.3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT-jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8.jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/lib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0.1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/storm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml-1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/storm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-time-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/conf:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/stormjar.jar
backtype.storm.daemon.worker fsd-1-1449794574
1953edf9-8f3e-4bd8-870a-05370272b097 6703
ff8603ef-5fb9-47c7-90d0-01813ace4188
user      3573  0.0  0.0  16016  2256 pts/28   S+   05:16   0:00 grep
--color=auto 2621


On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <sa...@gmail.com> wrote:

> please , how can i check it ?
>
> On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
>> Okay. Please check which application this process belongs to, and whether
>> is it OK to kill it.
>>
>> Regards
>> Basti
>> -----Original Message-----
>> From: sam mohel [mailto:sammohel5@gmail.com]
>> Sent: Friday, December 11, 2015 11:06 AM
>> To: dev@storm.apache.org
>> Subject: Re: Problem with storm since 4 months
>>
>> i used
>>
>>  netstat -ntape
>>
>> and got
>> Code:
>>
>> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
>> LISTEN      1000       22776       2621/java
>>
>> then used this command with PID of PORT = 2621
>> Code:
>>
>>  lsof -P -w -n -p 2621 -a -i
>>
>> and got
>> Code:
>>
>> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>> java    2621 user   76u  IPv4  18906      0t0  TCP
>> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
>> java    2621 user   83u  IPv4  22773      0t0  TCP
>> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
>> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
>>
>>
>> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
>> wrote:
>>
>> > Hi Sam,
>> >
>> > Have you tried "netstat" and "lsof" commands recommended by Erik? What
>> > is the result?
>> > We need to find which process bound this port, then kill the process
>> > to release this port.
>> >
>> > Regards
>> > Basti
>> >
>> > -----Original Message-----
>> > From: sam mohel [mailto:sammohel5@gmail.com]
>> > Sent: Friday, December 11, 2015 8:15 AM
>> > To: dev@storm.apache.org
>> > Subject: Re: Problem with storm since 4 months
>> >
>> > > now i converted the change
>> >
>> > but problem still
>> >
>> > > i mean in the link you sent , he said "the ephermal port range is
>> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably
>> > extend it to run from 16k to 64k."
>> >
>> > how can i do that ?
>> >
>> > > how can i check if there is anothe dump worker working ,
>> > or
>> > is there another way to kill any worker if it is dump or not
>> "completely"
>> > and start from scratch
>> >
>> > thanks for your patience and your time
>> >
>> >
>> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
>> > eweathers@groupon.com.invalid> wrote:
>> >
>> > > responses inline.
>> > >
>> > > >Ah, so, you shouldn't have touched the "local port range" setting
>> > > >since
>> > > you
>> > > don't fully understand ephemeral ports and TCP yet! :-)
>> > >
>> > > I'll give a brief synopsis:
>> > >
>> > > Say you're making a connection from a client to a server that
>> > > listens on port 80. The client needs to have a port of its own to
>> > > receive the response packets from the server. The port that is
>> > > allocated for it by the TCP stack is a "random" port, which is
>> > > called an "ephemeral" port in
>> > this context.
>> > > So with your original default config (32768    61000) the port would
>> be
>> > > chosen from an available port on the machine that lies within that
>> > > range of
>> > > 32768->61000.  That's good, it wouldn't conflict with the default
>> > > 32768->TCP ports
>> > > being listened to by the Storm Worker processes (67xx).  So 1. isn't
>> > > your problem.  But now you've made it possible for it to become a
>> > > problem, since now the range of ports that can be given as an
>> > > ephemeral port overlaps with the default Storm Worker ports
>> > > (1024->65535 includes 67xx).  So you should revert that config change.
>> > >
>> > > Thanks for your replying , now i reverted the change
>> >
>> > >
>> > > >Sorry, I don't understand what you are asking. What thing is "16"
>> > > >that
>> > > you
>> > > are trying to extend?
>> > >
>> > > i mean in the link you sent , he adviced to extend from 16 to 64
>> > > >You can kill topologies from the Nimbus UI (web page).  Or with the
>> > > bin/storm command.
>> > >
>> > >
>> > > >The commands are *solely* intended to figure out what is conflicting.
>> > > Your
>> > > logs claim that there is something holding onto 67xx which prevents
>> the
>> > > Storm Worker from launching.   So if that is happening you should
>> > > *immediately* try to figure out what is actually holding onto the
>> > > port and preventing your Storm Worker from launching.
>> > >
>> > >
>> > > >Not sure what you mean here.
>> > >
>> > > Please note that Storm often suffers from "cascading" failures,
>> > > where there are a lot of exceptions and errors that aren't actually
>> > > the root cause of the problem.  Often you need to spend time and
>> > > effort looking at lots of logs and tracing back to the real root
>> cause.
>> > >
>> > >
>> > >
>> > >
>> > > >
>> > > > Really , Thanks for your time
>> > > >
>> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
>> > > > eweathers@groupon.com.invalid> wrote:
>> > > >
>> > > > > Regarding Basti's suggestion (1.) that your host's configured
>> > > > > ephemeral ports might be conflicting with the storm worker
>> > > > > ports, here's how you
>> > > > can
>> > > > > check your "local port range" setting:
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how
>> > > -m any-ephemeral-ports-are-left-available
>> > > > >
>> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
>> > > > >
>> > > > >
>> > > > > It's possible that there is a zombie worker process holding onto
>> > > > > port
>> > > > 6703.
>> > > > > I would try to identify the process like so:
>> > > > >
>> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN |
>> > > > > grep -w
>> > > > 6703
>> > > > >
>> > > > > Alternatively you can try a global lsof search:
>> > > > >
>> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
>> > > > >
>> > > > > - Erik
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
>> > > basti.lj@alibaba-inc.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Sam,
>> > > > > >
>> > > > > > You can try to find which process has bound this port by
>> > > > > > "netstat
>> > > -anp"
>> > > > > > first.
>> > > > > >
>> > > > > > Generally, there are following cases for the binding error.
>> > > > > > 1. "local port range" is not set to exclude the port range
>> > > > > > used in
>> > > > Storm.
>> > > > > > 2. The previous worker was not killed correctly.
>> > > > > > 3. There is bug of assignment in some scenarios. Same port was
>> > > assigned
>> > > > > to
>> > > > > > two workers.
>> > > > > >
>> > > > > > Regards
>> > > > > > Basti
>> > > > > >
>> > > > > > -----Original Message-----
>> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
>> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
>> > > > > > To: dev@storm.apache.org
>> > > > > > Subject: Re: Problem with storm since 4 months
>> > > > > >
>> > > > > > i tried to use storm-0.9.5 but problem changed with
>> > > > > >
>> > > > > > cannot bind port 6703 i think it's same problem
>> > > > > >
>> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
>> > > > > >
>> > > > > > > Sam,
>> > > > > > >           you might be using very old version of storm since
>> > > > > > > its
>> > > > > showing
>> > > > > > >           ZeroMQ. Can you try using newer version storm
>> > > > > > > without
>> > > zero
>> > > > > mq.
>> > > > > > > -Harsha
>> > > > > > >
>> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
>> > > > > > > > I have this problem since 4months when I submitted
>> > > > > > > > topology I got this in the worker log file [ERROR] Async
>> loop died!
>> > > > > > org.zeromq.ZMQException:
>> > > > > > > > Address already in use(0x62) at
>> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
>> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
>> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
>> > > > > > > >
>> > > > > > >
>> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__16
>> > > > 29
>> > > > .i
>> > > > > > > nvoke(loader.clj:26)
>> > > > > > > > at
>> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375
>> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
>> > > > java.lang.Thread.run(Unknown
>> > > > > > > > Source)
>> > > > > > > >
>> > > > > > > > when i tried to connect port 6703 and 6702
>> > > > > > > >
>> > > > > > > > And supervisor log file hadn't still start
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I searched everywhere but cannot find any solution I hope
>> > > > > > > > you can
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>>
>>
>

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
please , how can i check it ?

On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Okay. Please check which application this process belongs to, and whether
> is it OK to kill it.
>
> Regards
> Basti
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Friday, December 11, 2015 11:06 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> i used
>
>  netstat -ntape
>
> and got
> Code:
>
> tcp        0      0 0.0.0.0:6703            0.0.0.0:*
> LISTEN      1000       22776       2621/java
>
> then used this command with PID of PORT = 2621
> Code:
>
>  lsof -P -w -n -p 2621 -a -i
>
> and got
> Code:
>
> COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
> java    2621 user   76u  IPv4  18906      0t0  TCP
> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
> java    2621 user   83u  IPv4  22773      0t0  TCP
> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
> java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)
>
>
> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
> > Hi Sam,
> >
> > Have you tried "netstat" and "lsof" commands recommended by Erik? What
> > is the result?
> > We need to find which process bound this port, then kill the process
> > to release this port.
> >
> > Regards
> > Basti
> >
> > -----Original Message-----
> > From: sam mohel [mailto:sammohel5@gmail.com]
> > Sent: Friday, December 11, 2015 8:15 AM
> > To: dev@storm.apache.org
> > Subject: Re: Problem with storm since 4 months
> >
> > > now i converted the change
> >
> > but problem still
> >
> > > i mean in the link you sent , he said "the ephermal port range is
> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably
> > extend it to run from 16k to 64k."
> >
> > how can i do that ?
> >
> > > how can i check if there is anothe dump worker working ,
> > or
> > is there another way to kill any worker if it is dump or not "completely"
> > and start from scratch
> >
> > thanks for your patience and your time
> >
> >
> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> > eweathers@groupon.com.invalid> wrote:
> >
> > > responses inline.
> > >
> > > >Ah, so, you shouldn't have touched the "local port range" setting
> > > >since
> > > you
> > > don't fully understand ephemeral ports and TCP yet! :-)
> > >
> > > I'll give a brief synopsis:
> > >
> > > Say you're making a connection from a client to a server that
> > > listens on port 80. The client needs to have a port of its own to
> > > receive the response packets from the server. The port that is
> > > allocated for it by the TCP stack is a "random" port, which is
> > > called an "ephemeral" port in
> > this context.
> > > So with your original default config (32768    61000) the port would be
> > > chosen from an available port on the machine that lies within that
> > > range of
> > > 32768->61000.  That's good, it wouldn't conflict with the default
> > > 32768->TCP ports
> > > being listened to by the Storm Worker processes (67xx).  So 1. isn't
> > > your problem.  But now you've made it possible for it to become a
> > > problem, since now the range of ports that can be given as an
> > > ephemeral port overlaps with the default Storm Worker ports
> > > (1024->65535 includes 67xx).  So you should revert that config change.
> > >
> > > Thanks for your replying , now i reverted the change
> >
> > >
> > > >Sorry, I don't understand what you are asking. What thing is "16"
> > > >that
> > > you
> > > are trying to extend?
> > >
> > > i mean in the link you sent , he adviced to extend from 16 to 64
> > > >You can kill topologies from the Nimbus UI (web page).  Or with the
> > > bin/storm command.
> > >
> > >
> > > >The commands are *solely* intended to figure out what is conflicting.
> > > Your
> > > logs claim that there is something holding onto 67xx which prevents the
> > > Storm Worker from launching.   So if that is happening you should
> > > *immediately* try to figure out what is actually holding onto the
> > > port and preventing your Storm Worker from launching.
> > >
> > >
> > > >Not sure what you mean here.
> > >
> > > Please note that Storm often suffers from "cascading" failures,
> > > where there are a lot of exceptions and errors that aren't actually
> > > the root cause of the problem.  Often you need to spend time and
> > > effort looking at lots of logs and tracing back to the real root cause.
> > >
> > >
> > >
> > >
> > > >
> > > > Really , Thanks for your time
> > > >
> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > > > eweathers@groupon.com.invalid> wrote:
> > > >
> > > > > Regarding Basti's suggestion (1.) that your host's configured
> > > > > ephemeral ports might be conflicting with the storm worker
> > > > > ports, here's how you
> > > > can
> > > > > check your "local port range" setting:
> > > > >
> > > > >
> > > > >
> > > >
> > > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how
> > > -m any-ephemeral-ports-are-left-available
> > > > >
> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > > > >
> > > > >
> > > > > It's possible that there is a zombie worker process holding onto
> > > > > port
> > > > 6703.
> > > > > I would try to identify the process like so:
> > > > >
> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN |
> > > > > grep -w
> > > > 6703
> > > > >
> > > > > Alternatively you can try a global lsof search:
> > > > >
> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > > > >
> > > > > - Erik
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > > basti.lj@alibaba-inc.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Sam,
> > > > > >
> > > > > > You can try to find which process has bound this port by
> > > > > > "netstat
> > > -anp"
> > > > > > first.
> > > > > >
> > > > > > Generally, there are following cases for the binding error.
> > > > > > 1. "local port range" is not set to exclude the port range
> > > > > > used in
> > > > Storm.
> > > > > > 2. The previous worker was not killed correctly.
> > > > > > 3. There is bug of assignment in some scenarios. Same port was
> > > assigned
> > > > > to
> > > > > > two workers.
> > > > > >
> > > > > > Regards
> > > > > > Basti
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > > > > To: dev@storm.apache.org
> > > > > > Subject: Re: Problem with storm since 4 months
> > > > > >
> > > > > > i tried to use storm-0.9.5 but problem changed with
> > > > > >
> > > > > > cannot bind port 6703 i think it's same problem
> > > > > >
> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > > > > >
> > > > > > > Sam,
> > > > > > >           you might be using very old version of storm since
> > > > > > > its
> > > > > showing
> > > > > > >           ZeroMQ. Can you try using newer version storm
> > > > > > > without
> > > zero
> > > > > mq.
> > > > > > > -Harsha
> > > > > > >
> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > > > > I have this problem since 4months when I submitted
> > > > > > > > topology I got this in the worker log file [ERROR] Async
> loop died!
> > > > > > org.zeromq.ZMQException:
> > > > > > > > Address already in use(0x62) at
> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > > > > >
> > > > > > >
> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__16
> > > > 29
> > > > .i
> > > > > > > nvoke(loader.clj:26)
> > > > > > > > at
> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375
> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> > > > java.lang.Thread.run(Unknown
> > > > > > > > Source)
> > > > > > > >
> > > > > > > > when i tried to connect port 6703 and 6702
> > > > > > > >
> > > > > > > > And supervisor log file hadn't still start
> > > > > > > >
> > > > > > > >
> > > > > > > > I searched everywhere but cannot find any solution I hope
> > > > > > > > you can
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
Okay. Please check which application this process belongs to, and whether is it OK to kill it.

Regards
Basti
-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Friday, December 11, 2015 11:06 AM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

i used

 netstat -ntape

and got
Code:

tcp        0      0 0.0.0.0:6703            0.0.0.0:*
LISTEN      1000       22776       2621/java

then used this command with PID of PORT = 2621
Code:

 lsof -P -w -n -p 2621 -a -i

and got
Code:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    2621 user   76u  IPv4  18906      0t0  TCP
127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
java    2621 user   83u  IPv4  22773      0t0  TCP
127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)


On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> Have you tried "netstat" and "lsof" commands recommended by Erik? What 
> is the result?
> We need to find which process bound this port, then kill the process 
> to release this port.
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Friday, December 11, 2015 8:15 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> > now i converted the change
>
> but problem still
>
> > i mean in the link you sent , he said "the ephermal port range is
> specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably 
> extend it to run from 16k to 64k."
>
> how can i do that ?
>
> > how can i check if there is anothe dump worker working ,
> or
> is there another way to kill any worker if it is dump or not "completely"
> and start from scratch
>
> thanks for your patience and your time
>
>
> On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < 
> eweathers@groupon.com.invalid> wrote:
>
> > responses inline.
> >
> > >Ah, so, you shouldn't have touched the "local port range" setting 
> > >since
> > you
> > don't fully understand ephemeral ports and TCP yet! :-)
> >
> > I'll give a brief synopsis:
> >
> > Say you're making a connection from a client to a server that 
> > listens on port 80. The client needs to have a port of its own to 
> > receive the response packets from the server. The port that is 
> > allocated for it by the TCP stack is a "random" port, which is 
> > called an "ephemeral" port in
> this context.
> > So with your original default config (32768    61000) the port would be
> > chosen from an available port on the machine that lies within that 
> > range of
> > 32768->61000.  That's good, it wouldn't conflict with the default 
> > 32768->TCP ports
> > being listened to by the Storm Worker processes (67xx).  So 1. isn't 
> > your problem.  But now you've made it possible for it to become a 
> > problem, since now the range of ports that can be given as an 
> > ephemeral port overlaps with the default Storm Worker ports
> > (1024->65535 includes 67xx).  So you should revert that config change.
> >
> > Thanks for your replying , now i reverted the change
>
> >
> > >Sorry, I don't understand what you are asking. What thing is "16"
> > >that
> > you
> > are trying to extend?
> >
> > i mean in the link you sent , he adviced to extend from 16 to 64
> > >You can kill topologies from the Nimbus UI (web page).  Or with the
> > bin/storm command.
> >
> >
> > >The commands are *solely* intended to figure out what is conflicting.
> > Your
> > logs claim that there is something holding onto 67xx which prevents the
> > Storm Worker from launching.   So if that is happening you should
> > *immediately* try to figure out what is actually holding onto the 
> > port and preventing your Storm Worker from launching.
> >
> >
> > >Not sure what you mean here.
> >
> > Please note that Storm often suffers from "cascading" failures, 
> > where there are a lot of exceptions and errors that aren't actually 
> > the root cause of the problem.  Often you need to spend time and 
> > effort looking at lots of logs and tracing back to the real root cause.
> >
> >
> >
> >
> > >
> > > Really , Thanks for your time
> > >
> > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < 
> > > eweathers@groupon.com.invalid> wrote:
> > >
> > > > Regarding Basti's suggestion (1.) that your host's configured 
> > > > ephemeral ports might be conflicting with the storm worker 
> > > > ports, here's how you
> > > can
> > > > check your "local port range" setting:
> > > >
> > > >
> > > >
> > >
> > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how
> > -m any-ephemeral-ports-are-left-available
> > > >
> > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > > >
> > > >
> > > > It's possible that there is a zombie worker process holding onto 
> > > > port
> > > 6703.
> > > > I would try to identify the process like so:
> > > >
> > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | 
> > > > grep -w
> > > 6703
> > > >
> > > > Alternatively you can try a global lsof search:
> > > >
> > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > > >
> > > > - Erik
> > > >
> > > >
> > > >
> > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > basti.lj@alibaba-inc.com>
> > > > wrote:
> > > >
> > > > > Hi Sam,
> > > > >
> > > > > You can try to find which process has bound this port by 
> > > > > "netstat
> > -anp"
> > > > > first.
> > > > >
> > > > > Generally, there are following cases for the binding error.
> > > > > 1. "local port range" is not set to exclude the port range 
> > > > > used in
> > > Storm.
> > > > > 2. The previous worker was not killed correctly.
> > > > > 3. There is bug of assignment in some scenarios. Same port was
> > assigned
> > > > to
> > > > > two workers.
> > > > >
> > > > > Regards
> > > > > Basti
> > > > >
> > > > > -----Original Message-----
> > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > > > To: dev@storm.apache.org
> > > > > Subject: Re: Problem with storm since 4 months
> > > > >
> > > > > i tried to use storm-0.9.5 but problem changed with
> > > > >
> > > > > cannot bind port 6703 i think it's same problem
> > > > >
> > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > > > >
> > > > > > Sam,
> > > > > >           you might be using very old version of storm since 
> > > > > > its
> > > > showing
> > > > > >           ZeroMQ. Can you try using newer version storm 
> > > > > > without
> > zero
> > > > mq.
> > > > > > -Harsha
> > > > > >
> > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > > > I have this problem since 4months when I submitted 
> > > > > > > topology I got this in the worker log file [ERROR] Async loop died!
> > > > > org.zeromq.ZMQException:
> > > > > > > Address already in use(0x62) at 
> > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > > > zilch.mq$bind.invoke(mq.clj:69) at 
> > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > > > >
> > > > > >
> > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__16
> > > 29
> > > .i
> > > > > > nvoke(loader.clj:26)
> > > > > > > at
> > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375
> > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at
> > > java.lang.Thread.run(Unknown
> > > > > > > Source)
> > > > > > >
> > > > > > > when i tried to connect port 6703 and 6702
> > > > > > >
> > > > > > > And supervisor log file hadn't still start
> > > > > > >
> > > > > > >
> > > > > > > I searched everywhere but cannot find any solution I hope 
> > > > > > > you can
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
i used

 netstat -ntape

and got
Code:

tcp        0      0 0.0.0.0:6703            0.0.0.0:*
LISTEN      1000       22776       2621/java

then used this command with PID of PORT = 2621
Code:

 lsof -P -w -n -p 2621 -a -i

and got
Code:

COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    2621 user   76u  IPv4  18906      0t0  TCP
127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED)
java    2621 user   83u  IPv4  22773      0t0  TCP
127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED)
java    2621 user   86u  IPv4  22776      0t0  TCP *:6703 (LISTEN)


On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> Have you tried "netstat" and "lsof" commands recommended by Erik? What is
> the result?
> We need to find which process bound this port, then kill the process to
> release this port.
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Friday, December 11, 2015 8:15 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> > now i converted the change
>
> but problem still
>
> > i mean in the link you sent , he said "the ephermal port range is
> specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably
> extend it to run from 16k to 64k."
>
> how can i do that ?
>
> > how can i check if there is anothe dump worker working ,
> or
> is there another way to kill any worker if it is dump or not "completely"
> and start from scratch
>
> thanks for your patience and your time
>
>
> On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
> eweathers@groupon.com.invalid> wrote:
>
> > responses inline.
> >
> > >Ah, so, you shouldn't have touched the "local port range" setting
> > >since
> > you
> > don't fully understand ephemeral ports and TCP yet! :-)
> >
> > I'll give a brief synopsis:
> >
> > Say you're making a connection from a client to a server that listens
> > on port 80. The client needs to have a port of its own to receive the
> > response packets from the server. The port that is allocated for it by
> > the TCP stack is a "random" port, which is called an "ephemeral" port in
> this context.
> > So with your original default config (32768    61000) the port would be
> > chosen from an available port on the machine that lies within that
> > range of
> > 32768->61000.  That's good, it wouldn't conflict with the default TCP
> > 32768->ports
> > being listened to by the Storm Worker processes (67xx).  So 1. isn't
> > your problem.  But now you've made it possible for it to become a
> > problem, since now the range of ports that can be given as an
> > ephemeral port overlaps with the default Storm Worker ports
> > (1024->65535 includes 67xx).  So you should revert that config change.
> >
> > Thanks for your replying , now i reverted the change
>
> >
> > >Sorry, I don't understand what you are asking. What thing is "16"
> > >that
> > you
> > are trying to extend?
> >
> > i mean in the link you sent , he adviced to extend from 16 to 64
> > >You can kill topologies from the Nimbus UI (web page).  Or with the
> > bin/storm command.
> >
> >
> > >The commands are *solely* intended to figure out what is conflicting.
> > Your
> > logs claim that there is something holding onto 67xx which prevents the
> > Storm Worker from launching.   So if that is happening you should
> > *immediately* try to figure out what is actually holding onto the port
> > and preventing your Storm Worker from launching.
> >
> >
> > >Not sure what you mean here.
> >
> > Please note that Storm often suffers from "cascading" failures, where
> > there are a lot of exceptions and errors that aren't actually the root
> > cause of the problem.  Often you need to spend time and effort looking
> > at lots of logs and tracing back to the real root cause.
> >
> >
> >
> >
> > >
> > > Really , Thanks for your time
> > >
> > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > > eweathers@groupon.com.invalid> wrote:
> > >
> > > > Regarding Basti's suggestion (1.) that your host's configured
> > > > ephemeral ports might be conflicting with the storm worker ports,
> > > > here's how you
> > > can
> > > > check your "local port range" setting:
> > > >
> > > >
> > > >
> > >
> > https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-m
> > any-ephemeral-ports-are-left-available
> > > >
> > > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > > >
> > > >
> > > > It's possible that there is a zombie worker process holding onto
> > > > port
> > > 6703.
> > > > I would try to identify the process like so:
> > > >
> > > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN |
> > > > grep -w
> > > 6703
> > > >
> > > > Alternatively you can try a global lsof search:
> > > >
> > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > > >
> > > > - Erik
> > > >
> > > >
> > > >
> > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> > basti.lj@alibaba-inc.com>
> > > > wrote:
> > > >
> > > > > Hi Sam,
> > > > >
> > > > > You can try to find which process has bound this port by
> > > > > "netstat
> > -anp"
> > > > > first.
> > > > >
> > > > > Generally, there are following cases for the binding error.
> > > > > 1. "local port range" is not set to exclude the port range used
> > > > > in
> > > Storm.
> > > > > 2. The previous worker was not killed correctly.
> > > > > 3. There is bug of assignment in some scenarios. Same port was
> > assigned
> > > > to
> > > > > two workers.
> > > > >
> > > > > Regards
> > > > > Basti
> > > > >
> > > > > -----Original Message-----
> > > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > > > To: dev@storm.apache.org
> > > > > Subject: Re: Problem with storm since 4 months
> > > > >
> > > > > i tried to use storm-0.9.5 but problem changed with
> > > > >
> > > > > cannot bind port 6703 i think it's same problem
> > > > >
> > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > > > >
> > > > > > Sam,
> > > > > >           you might be using very old version of storm since
> > > > > > its
> > > > showing
> > > > > >           ZeroMQ. Can you try using newer version storm
> > > > > > without
> > zero
> > > > mq.
> > > > > > -Harsha
> > > > > >
> > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > > > I have this problem since 4months when I submitted topology
> > > > > > > I got this in the worker log file [ERROR] Async loop died!
> > > > > org.zeromq.ZMQException:
> > > > > > > Address already in use(0x62) at
> > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > > > >
> > > > > >
> > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629
> > > .i
> > > > > > nvoke(loader.clj:26)
> > > > > > > at
> > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > > > > > at clojure.lang.AFn.run(AFn.java:24) at
> > > java.lang.Thread.run(Unknown
> > > > > > > Source)
> > > > > > >
> > > > > > > when i tried to connect port 6703 and 6702
> > > > > > >
> > > > > > > And supervisor log file hadn't still start
> > > > > > >
> > > > > > >
> > > > > > > I searched everywhere but cannot find any solution I hope
> > > > > > > you can
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
Hi Sam,

Have you tried "netstat" and "lsof" commands recommended by Erik? What is the result?
We need to find which process bound this port, then kill the process to release this port.

Regards
Basti

-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Friday, December 11, 2015 8:15 AM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

> now i converted the change

but problem still

> i mean in the link you sent , he said "the ephermal port range is
specified in /proc/sys/net/ipv4/ip_local_ port_range. You can probably extend it to run from 16k to 64k."

how can i do that ?

> how can i check if there is anothe dump worker working ,
or
is there another way to kill any worker if it is dump or not "completely"
and start from scratch

thanks for your patience and your time


On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < eweathers@groupon.com.invalid> wrote:

> responses inline.
>
> >Ah, so, you shouldn't have touched the "local port range" setting 
> >since
> you
> don't fully understand ephemeral ports and TCP yet! :-)
>
> I'll give a brief synopsis:
>
> Say you're making a connection from a client to a server that listens 
> on port 80. The client needs to have a port of its own to receive the 
> response packets from the server. The port that is allocated for it by 
> the TCP stack is a "random" port, which is called an "ephemeral" port in this context.
> So with your original default config (32768    61000) the port would be
> chosen from an available port on the machine that lies within that 
> range of
> 32768->61000.  That's good, it wouldn't conflict with the default TCP 
> 32768->ports
> being listened to by the Storm Worker processes (67xx).  So 1. isn't 
> your problem.  But now you've made it possible for it to become a 
> problem, since now the range of ports that can be given as an 
> ephemeral port overlaps with the default Storm Worker ports 
> (1024->65535 includes 67xx).  So you should revert that config change.
>
> Thanks for your replying , now i reverted the change

>
> >Sorry, I don't understand what you are asking. What thing is "16" 
> >that
> you
> are trying to extend?
>
> i mean in the link you sent , he adviced to extend from 16 to 64
> >You can kill topologies from the Nimbus UI (web page).  Or with the
> bin/storm command.
>
>
> >The commands are *solely* intended to figure out what is conflicting.
> Your
> logs claim that there is something holding onto 67xx which prevents the
> Storm Worker from launching.   So if that is happening you should
> *immediately* try to figure out what is actually holding onto the port 
> and preventing your Storm Worker from launching.
>
>
> >Not sure what you mean here.
>
> Please note that Storm often suffers from "cascading" failures, where 
> there are a lot of exceptions and errors that aren't actually the root 
> cause of the problem.  Often you need to spend time and effort looking 
> at lots of logs and tracing back to the real root cause.
>
>
>
>
> >
> > Really , Thanks for your time
> >
> > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < 
> > eweathers@groupon.com.invalid> wrote:
> >
> > > Regarding Basti's suggestion (1.) that your host's configured 
> > > ephemeral ports might be conflicting with the storm worker ports, 
> > > here's how you
> > can
> > > check your "local port range" setting:
> > >
> > >
> > >
> >
> https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-m
> any-ephemeral-ports-are-left-available
> > >
> > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > >
> > >
> > > It's possible that there is a zombie worker process holding onto 
> > > port
> > 6703.
> > > I would try to identify the process like so:
> > >
> > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | 
> > > grep -w
> > 6703
> > >
> > > Alternatively you can try a global lsof search:
> > >
> > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > >
> > > - Erik
> > >
> > >
> > >
> > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> basti.lj@alibaba-inc.com>
> > > wrote:
> > >
> > > > Hi Sam,
> > > >
> > > > You can try to find which process has bound this port by 
> > > > "netstat
> -anp"
> > > > first.
> > > >
> > > > Generally, there are following cases for the binding error.
> > > > 1. "local port range" is not set to exclude the port range used 
> > > > in
> > Storm.
> > > > 2. The previous worker was not killed correctly.
> > > > 3. There is bug of assignment in some scenarios. Same port was
> assigned
> > > to
> > > > two workers.
> > > >
> > > > Regards
> > > > Basti
> > > >
> > > > -----Original Message-----
> > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > > To: dev@storm.apache.org
> > > > Subject: Re: Problem with storm since 4 months
> > > >
> > > > i tried to use storm-0.9.5 but problem changed with
> > > >
> > > > cannot bind port 6703 i think it's same problem
> > > >
> > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > > >
> > > > > Sam,
> > > > >           you might be using very old version of storm since 
> > > > > its
> > > showing
> > > > >           ZeroMQ. Can you try using newer version storm 
> > > > > without
> zero
> > > mq.
> > > > > -Harsha
> > > > >
> > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > > I have this problem since 4months when I submitted topology 
> > > > > > I got this in the worker log file [ERROR] Async loop died!
> > > > org.zeromq.ZMQException:
> > > > > > Address already in use(0x62) at 
> > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > > zilch.mq$bind.invoke(mq.clj:69) at 
> > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > > >
> > > > >
> > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629
> > .i
> > > > > nvoke(loader.clj:26)
> > > > > > at 
> > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > > > > at clojure.lang.AFn.run(AFn.java:24) at
> > java.lang.Thread.run(Unknown
> > > > > > Source)
> > > > > >
> > > > > > when i tried to connect port 6703 and 6702
> > > > > >
> > > > > > And supervisor log file hadn't still start
> > > > > >
> > > > > >
> > > > > > I searched everywhere but cannot find any solution I hope 
> > > > > > you can
> > > > >
> > > >
> > > >
> > >
> >
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
> now i converted the change

but problem still

> i mean in the link you sent , he said "the ephermal port range is
specified in /proc/sys/net/ipv4/ip_local_
port_range. You can probably extend it to run from 16k to 64k."

how can i do that ?

> how can i check if there is anothe dump worker working ,
or
is there another way to kill any worker if it is dump or not "completely"
and start from scratch

thanks for your patience and your time


On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers <
eweathers@groupon.com.invalid> wrote:

> responses inline.
>
> >Ah, so, you shouldn't have touched the "local port range" setting since
> you
> don't fully understand ephemeral ports and TCP yet! :-)
>
> I'll give a brief synopsis:
>
> Say you're making a connection from a client to a server that listens on
> port 80. The client needs to have a port of its own to receive the response
> packets from the server. The port that is allocated for it by the TCP stack
> is a "random" port, which is called an "ephemeral" port in this context.
> So with your original default config (32768    61000) the port would be
> chosen from an available port on the machine that lies within that range of
> 32768->61000.  That's good, it wouldn't conflict with the default TCP ports
> being listened to by the Storm Worker processes (67xx).  So 1. isn't your
> problem.  But now you've made it possible for it to become a problem, since
> now the range of ports that can be given as an ephemeral port overlaps with
> the default Storm Worker ports (1024->65535 includes 67xx).  So you should
> revert that config change.
>
> Thanks for your replying , now i reverted the change

>
> >Sorry, I don't understand what you are asking. What thing is "16" that
> you
> are trying to extend?
>
> i mean in the link you sent , he adviced to extend from 16 to 64
> >You can kill topologies from the Nimbus UI (web page).  Or with the
> bin/storm command.
>
>
> >The commands are *solely* intended to figure out what is conflicting.
> Your
> logs claim that there is something holding onto 67xx which prevents the
> Storm Worker from launching.   So if that is happening you should
> *immediately* try to figure out what is actually holding onto the port and
> preventing your Storm Worker from launching.
>
>
> >Not sure what you mean here.
>
> Please note that Storm often suffers from "cascading" failures, where there
> are a lot of exceptions and errors that aren't actually the root cause of
> the problem.  Often you need to spend time and effort looking at lots of
> logs and tracing back to the real root cause.
>
>
>
>
> >
> > Really , Thanks for your time
> >
> > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> > eweathers@groupon.com.invalid> wrote:
> >
> > > Regarding Basti's suggestion (1.) that your host's configured ephemeral
> > > ports might be conflicting with the storm worker ports, here's how you
> > can
> > > check your "local port range" setting:
> > >
> > >
> > >
> >
> https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-many-ephemeral-ports-are-left-available
> > >
> > > % cat /proc/sys/net/ipv4/ip_local_port_range
> > >
> > >
> > > It's possible that there is a zombie worker process holding onto port
> > 6703.
> > > I would try to identify the process like so:
> > >
> > > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | grep -w
> > 6703
> > >
> > > Alternatively you can try a global lsof search:
> > >
> > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> > >
> > > - Erik
> > >
> > >
> > >
> > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <
> basti.lj@alibaba-inc.com>
> > > wrote:
> > >
> > > > Hi Sam,
> > > >
> > > > You can try to find which process has bound this port by "netstat
> -anp"
> > > > first.
> > > >
> > > > Generally, there are following cases for the binding error.
> > > > 1. "local port range" is not set to exclude the port range used in
> > Storm.
> > > > 2. The previous worker was not killed correctly.
> > > > 3. There is bug of assignment in some scenarios. Same port was
> assigned
> > > to
> > > > two workers.
> > > >
> > > > Regards
> > > > Basti
> > > >
> > > > -----Original Message-----
> > > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > > Sent: Thursday, December 10, 2015 7:16 AM
> > > > To: dev@storm.apache.org
> > > > Subject: Re: Problem with storm since 4 months
> > > >
> > > > i tried to use storm-0.9.5 but problem changed with
> > > >
> > > > cannot bind port 6703 i think it's same problem
> > > >
> > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > > >
> > > > > Sam,
> > > > >           you might be using very old version of storm since its
> > > showing
> > > > >           ZeroMQ. Can you try using newer version storm without
> zero
> > > mq.
> > > > > -Harsha
> > > > >
> > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > > I have this problem since 4months when I submitted topology I got
> > > > > > this in the worker log file [ERROR] Async loop died!
> > > > org.zeromq.ZMQException:
> > > > > > Address already in use(0x62)
> > > > > > at org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > > >
> > > > >
> > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> > > > > nvoke(loader.clj:26)
> > > > > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > > > > at clojure.lang.AFn.run(AFn.java:24) at
> > java.lang.Thread.run(Unknown
> > > > > > Source)
> > > > > >
> > > > > > when i tried to connect port 6703 and 6702
> > > > > >
> > > > > > And supervisor log file hadn't still start
> > > > > >
> > > > > >
> > > > > > I searched everywhere but cannot find any solution I hope you can
> > > > >
> > > >
> > > >
> > >
> >
>

Re: Problem with storm since 4 months

Posted by Erik Weathers <ew...@groupon.com.INVALID>.
responses inline.

On Thu, Dec 10, 2015 at 4:48 AM, sam mohel <sa...@gmail.com> wrote:

> Thanks for your replying and for the link
> the local port range of my machine default is 32768    61000 and i changed
> it to 1024    65535 is that right or not ?
>

Ah, so, you shouldn't have touched the "local port range" setting since you
don't fully understand ephemeral ports and TCP yet! :-)

I'll give a brief synopsis:

Say you're making a connection from a client to a server that listens on
port 80. The client needs to have a port of its own to receive the response
packets from the server. The port that is allocated for it by the TCP stack
is a "random" port, which is called an "ephemeral" port in this context.
So with your original default config (32768    61000) the port would be
chosen from an available port on the machine that lies within that range of
32768->61000.  That's good, it wouldn't conflict with the default TCP ports
being listened to by the Storm Worker processes (67xx).  So 1. isn't your
problem.  But now you've made it possible for it to become a problem, since
now the range of ports that can be given as an ephemeral port overlaps with
the default Storm Worker ports (1024->65535 includes 67xx).  So you should
revert that config change.


>
> how can i extend from 16 to 64 ? i searched to know how but didn't get it
>

Sorry, I don't understand what you are asking. What thing is "16" that you
are trying to extend?


> Now i was running storm and submitted topology but electricity felled so
> how can i killed the topology ? am i connect storm again then kill or what
> shoud i do ?
>

You can kill topologies from the Nimbus UI (web page).  Or with the
bin/storm command.


>
> should i try your commands after submitting or before it also ?
>

The commands are *solely* intended to figure out what is conflicting.  Your
logs claim that there is something holding onto 67xx which prevents the
Storm Worker from launching.   So if that is happening you should
*immediately* try to figure out what is actually holding onto the port and
preventing your Storm Worker from launching.


>
> i feel that problem is really with local port but can't catch it
>

Not sure what you mean here.

Please note that Storm often suffers from "cascading" failures, where there
are a lot of exceptions and errors that aren't actually the root cause of
the problem.  Often you need to spend time and effort looking at lots of
logs and tracing back to the real root cause.

- Erik


>
> Really , Thanks for your time
>
> On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
> eweathers@groupon.com.invalid> wrote:
>
> > Regarding Basti's suggestion (1.) that your host's configured ephemeral
> > ports might be conflicting with the storm worker ports, here's how you
> can
> > check your "local port range" setting:
> >
> >
> >
> https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-many-ephemeral-ports-are-left-available
> >
> > % cat /proc/sys/net/ipv4/ip_local_port_range
> >
> >
> > It's possible that there is a zombie worker process holding onto port
> 6703.
> > I would try to identify the process like so:
> >
> > % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | grep -w
> 6703
> >
> > Alternatively you can try a global lsof search:
> >
> > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
> >
> > - Erik
> >
> >
> >
> > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> > wrote:
> >
> > > Hi Sam,
> > >
> > > You can try to find which process has bound this port by "netstat -anp"
> > > first.
> > >
> > > Generally, there are following cases for the binding error.
> > > 1. "local port range" is not set to exclude the port range used in
> Storm.
> > > 2. The previous worker was not killed correctly.
> > > 3. There is bug of assignment in some scenarios. Same port was assigned
> > to
> > > two workers.
> > >
> > > Regards
> > > Basti
> > >
> > > -----Original Message-----
> > > From: sam mohel [mailto:sammohel5@gmail.com]
> > > Sent: Thursday, December 10, 2015 7:16 AM
> > > To: dev@storm.apache.org
> > > Subject: Re: Problem with storm since 4 months
> > >
> > > i tried to use storm-0.9.5 but problem changed with
> > >
> > > cannot bind port 6703 i think it's same problem
> > >
> > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> > >
> > > > Sam,
> > > >           you might be using very old version of storm since its
> > showing
> > > >           ZeroMQ. Can you try using newer version storm without zero
> > mq.
> > > > -Harsha
> > > >
> > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > > I have this problem since 4months when I submitted topology I got
> > > > > this in the worker log file [ERROR] Async loop died!
> > > org.zeromq.ZMQException:
> > > > > Address already in use(0x62)
> > > > > at org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > > >
> > > >
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> > > > nvoke(loader.clj:26)
> > > > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > > > at clojure.lang.AFn.run(AFn.java:24) at
> java.lang.Thread.run(Unknown
> > > > > Source)
> > > > >
> > > > > when i tried to connect port 6703 and 6702
> > > > >
> > > > > And supervisor log file hadn't still start
> > > > >
> > > > >
> > > > > I searched everywhere but cannot find any solution I hope you can
> > > >
> > >
> > >
> >
>

Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
Thanks for your replying and for the link
the local port range of my machine default is 32768    61000 and i changed
it to 1024    65535 is that right or not ?

how can i extend from 16 to 64 ? i searched to know how but didn't get it

Now i was running storm and submitted topology but electricity felled so
how can i killed the topology ? am i connect storm again then kill or what
shoud i do ?

should i try your commands after submitting or before it also ?

i feel that problem is really with local port but can't catch it

Really , Thanks for your time

On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers <
eweathers@groupon.com.invalid> wrote:

> Regarding Basti's suggestion (1.) that your host's configured ephemeral
> ports might be conflicting with the storm worker ports, here's how you can
> check your "local port range" setting:
>
>
> https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-many-ephemeral-ports-are-left-available
>
> % cat /proc/sys/net/ipv4/ip_local_port_range
>
>
> It's possible that there is a zombie worker process holding onto port 6703.
> I would try to identify the process like so:
>
> % sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | grep -w 6703
>
> Alternatively you can try a global lsof search:
>
> % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703
>
> - Erik
>
>
>
> On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
> wrote:
>
> > Hi Sam,
> >
> > You can try to find which process has bound this port by "netstat -anp"
> > first.
> >
> > Generally, there are following cases for the binding error.
> > 1. "local port range" is not set to exclude the port range used in Storm.
> > 2. The previous worker was not killed correctly.
> > 3. There is bug of assignment in some scenarios. Same port was assigned
> to
> > two workers.
> >
> > Regards
> > Basti
> >
> > -----Original Message-----
> > From: sam mohel [mailto:sammohel5@gmail.com]
> > Sent: Thursday, December 10, 2015 7:16 AM
> > To: dev@storm.apache.org
> > Subject: Re: Problem with storm since 4 months
> >
> > i tried to use storm-0.9.5 but problem changed with
> >
> > cannot bind port 6703 i think it's same problem
> >
> > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
> >
> > > Sam,
> > >           you might be using very old version of storm since its
> showing
> > >           ZeroMQ. Can you try using newer version storm without zero
> mq.
> > > -Harsha
> > >
> > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > > I have this problem since 4months when I submitted topology I got
> > > > this in the worker log file [ERROR] Async loop died!
> > org.zeromq.ZMQException:
> > > > Address already in use(0x62)
> > > > at org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > > zilch.mq$bind.invoke(mq.clj:69) at
> > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > > >
> > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> > > nvoke(loader.clj:26)
> > > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > > at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> > > > Source)
> > > >
> > > > when i tried to connect port 6703 and 6702
> > > >
> > > > And supervisor log file hadn't still start
> > > >
> > > >
> > > > I searched everywhere but cannot find any solution I hope you can
> > >
> >
> >
>

Re: Problem with storm since 4 months

Posted by Erik Weathers <ew...@groupon.com.INVALID>.
Regarding Basti's suggestion (1.) that your host's configured ephemeral
ports might be conflicting with the storm worker ports, here's how you can
check your "local port range" setting:

https://serverfault.com/questions/261663/on-linux-how-can-i-tell-how-many-ephemeral-ports-are-left-available

% cat /proc/sys/net/ipv4/ip_local_port_range


It's possible that there is a zombie worker process holding onto port 6703.
I would try to identify the process like so:

% sudo netstat -ap --numeric-ports --extend | grep -w LISTEN | grep -w 6703

Alternatively you can try a global lsof search:

% sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703

- Erik



On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) <ba...@alibaba-inc.com>
wrote:

> Hi Sam,
>
> You can try to find which process has bound this port by "netstat -anp"
> first.
>
> Generally, there are following cases for the binding error.
> 1. "local port range" is not set to exclude the port range used in Storm.
> 2. The previous worker was not killed correctly.
> 3. There is bug of assignment in some scenarios. Same port was assigned to
> two workers.
>
> Regards
> Basti
>
> -----Original Message-----
> From: sam mohel [mailto:sammohel5@gmail.com]
> Sent: Thursday, December 10, 2015 7:16 AM
> To: dev@storm.apache.org
> Subject: Re: Problem with storm since 4 months
>
> i tried to use storm-0.9.5 but problem changed with
>
> cannot bind port 6703 i think it's same problem
>
> On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:
>
> > Sam,
> >           you might be using very old version of storm since its showing
> >           ZeroMQ. Can you try using newer version storm without zero mq.
> > -Harsha
> >
> > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > > I have this problem since 4months when I submitted topology I got
> > > this in the worker log file [ERROR] Async loop died!
> org.zeromq.ZMQException:
> > > Address already in use(0x62)
> > > at org.zeromq.ZMQ$Socket.bind(Native Method) at
> > > zilch.mq$bind.invoke(mq.clj:69) at
> > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> > >
> > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> > nvoke(loader.clj:26)
> > > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > > at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> > > Source)
> > >
> > > when i tried to connect port 6703 and 6702
> > >
> > > And supervisor log file hadn't still start
> > >
> > >
> > > I searched everywhere but cannot find any solution I hope you can
> >
>
>

RE: Problem with storm since 4 months

Posted by "刘键(Basti Liu)" <ba...@alibaba-inc.com>.
Hi Sam,

You can try to find which process has bound this port by "netstat -anp" first.

Generally, there are following cases for the binding error.
1. "local port range" is not set to exclude the port range used in Storm.
2. The previous worker was not killed correctly.
3. There is bug of assignment in some scenarios. Same port was assigned to two workers.

Regards
Basti

-----Original Message-----
From: sam mohel [mailto:sammohel5@gmail.com] 
Sent: Thursday, December 10, 2015 7:16 AM
To: dev@storm.apache.org
Subject: Re: Problem with storm since 4 months

i tried to use storm-0.9.5 but problem changed with

cannot bind port 6703 i think it's same problem

On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:

> Sam,
>           you might be using very old version of storm since its showing
>           ZeroMQ. Can you try using newer version storm without zero mq.
> -Harsha
>
> On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > I have this problem since 4months when I submitted topology I got 
> > this in the worker log file [ERROR] Async loop died! org.zeromq.ZMQException:
> > Address already in use(0x62)
> > at org.zeromq.ZMQ$Socket.bind(Native Method) at 
> > zilch.mq$bind.invoke(mq.clj:69) at 
> > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> >
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.i
> nvoke(loader.clj:26)
> > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> > Source)
> >
> > when i tried to connect port 6703 and 6702
> >
> > And supervisor log file hadn't still start
> >
> >
> > I searched everywhere but cannot find any solution I hope you can
>


Re: Problem with storm since 4 months

Posted by sam mohel <sa...@gmail.com>.
i tried to use storm-0.9.5 but problem changed with

cannot bind port 6703 i think it's same problem

On Wed, Dec 9, 2015 at 8:42 PM, Harsha <st...@harsha.io> wrote:

> Sam,
>           you might be using very old version of storm since its showing
>           ZeroMQ. Can you try using newer version storm without zero mq.
> -Harsha
>
> On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> > I have this problem since 4months when I submitted topology I got this in
> > the worker log file [ERROR] Async loop died! org.zeromq.ZMQException:
> > Address already in use(0x62)
> > at org.zeromq.ZMQ$Socket.bind(Native Method)
> > at zilch.mq$bind.invoke(mq.clj:69)
> > at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> >
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> > at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> > Source)
> >
> > when i tried to connect port 6703 and 6702
> >
> > And supervisor log file hadn't still start
> >
> >
> > I searched everywhere but cannot find any solution I hope you can
>

Re: Problem with storm since 4 months

Posted by Harsha <st...@harsha.io>.
Sam,
          you might be using very old version of storm since its showing
          ZeroMQ. Can you try using newer version storm without zero mq.
-Harsha 

On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote:
> I have this problem since 4months when I submitted topology I got this in
> the worker log file [ERROR] Async loop died! org.zeromq.ZMQException:
> Address already in use(0x62)
> at org.zeromq.ZMQ$Socket.bind(Native Method)
> at zilch.mq$bind.invoke(mq.clj:69)
> at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57)at
> backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26)
> at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)
> at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Unknown
> Source)
> 
> when i tried to connect port 6703 and 6702
> 
> And supervisor log file hadn't still start
> 
> 
> I searched everywhere but cannot find any solution I hope you can