You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Nikita <ni...@gmail.com> on 2022/12/02 16:25:01 UTC

Address already in use issue

Hi, I have been facing Address already in use cycle error for a long period of
time. I saw the most common answer that it means that something is listening
on that port. But as I understood from logs after storm asks for rebalancing
(by some reason when storm gets a new leader assigned) or you deploy\redeploy
topologies it happens. But its not other process is listening on port its
storm topology listening on that port. Supervisor launch a topology (for
example 2:15:00) and after some time (based on first heartbeat (at 2:15:05)
change status of a slot from waiting-for-worker-start to kill-blob-update by
unknown for me reason and starts cleaning paths from storm/workers/id and
storm/workers-users/topology_id, Removed Worker ID and so on and finished. But
after few seconds (for example at 2:15:10) worker created that instance. And
what we have, supervisor deleted that worker id and don’t know already about
the topology and thinks it was removed but actually worker created that
topology instance and all calls of a supervisor after to create a topology
fail because supervisor sees that port is free but actually its not. (you can
read full explanations by me, log samples, config here
https://stackoverflow.com/questions/74345286/apache-storm-address-already-in-
use-error).  
I may be wrong but why I am sure it’s a topology is listening on port:  
1) I see a folder with worker id in storm/workers with creating date (and last
update) at +- deploying time.  
2) Same related to workers-artifacts\EventHandler-3-1669745351\6700
(EventHandler-3-1669745351 it’s a name + id of a problem topology)  
3)I can fix it either by stopping storm, deleting storm/ folder and cleaning
zookeeper or just by removing that worker id folder manually on supervisor
machine and after that exception disappears. But its pretty annoying and hard
to find fast a folder, for prod it may be critical.  
  
I would be very thankful to you if you help me to solve that one of annoying
storm issues. I couldn’t find a conf to add some delay to it.  
  
additional info:  
logs attached, busy port if 6700, EventHandler-3-1669745351 is a topology
name+id, worker id 1d375d63-38e3-4b6d-8d9e-4db7dd9bc20e



Sent from [Mail](https://go.microsoft.com/fwlink/?LinkId=550986) for Windows




Re: Address already in use issue

Posted by Mykyta Piddubskiy <ni...@gmail.com>.
Sorry one remark that in order to fix it I don't need to delete anything
from folders (but maybe I can) I need to just kill pid but its anyway java
process and I am 100% sure its storm topology.

пт, 2 дек. 2022 г. в 18:25, Nikita <ni...@gmail.com>:

> Hi, I have been facing Address already in use cycle error for a long
> period of time. I saw the most common answer that it means that something
> is listening on that port. But as I understood from logs after storm asks
> for rebalancing (by some reason when storm gets a new leader assigned) or
> you deploy\redeploy topologies it happens. But its not other process is
> listening on port its storm topology listening on that port. Supervisor
> launch a topology (for example 2:15:00) and after some time (based on first
> heartbeat (at 2:15:05) change status of a slot from
> waiting-for-worker-start to kill-blob-update by unknown for me reason and
> starts cleaning paths from storm/workers/id and
> storm/workers-users/topology_id, Removed Worker ID and so on and finished.
> But after few seconds (for example at 2:15:10) worker created that
> instance. And what we have, supervisor deleted that worker id and don’t
> know already about the topology and thinks it was removed but actually
> worker created that topology instance and all calls of a supervisor after
> to create a topology fail because supervisor sees that port is free but
> actually its not.   (you can read full explanations by me, log samples,
> config here
> https://stackoverflow.com/questions/74345286/apache-storm-address-already-in-use-error
> ).
> I may be wrong but why I am sure it’s a topology is listening on port:
> 1) I see a folder with worker id in storm/workers with creating date (and
> last update) at +- deploying time.
> 2) Same related to workers-artifacts\EventHandler-3-1669745351\6700
> (EventHandler-3-1669745351 it’s a name + id of a problem topology)
> 3)I can fix it either by stopping storm, deleting storm/ folder and
> cleaning zookeeper or just by removing that worker id folder manually on
> supervisor machine and after that exception disappears. But its pretty
> annoying and hard to find fast a folder, for prod it may be critical.
>
> I would be very thankful to you if you help me to solve that one of
> annoying storm issues. I couldn’t find a conf to add some delay to it.
>
> additional info:
> logs attached, busy port if 6700, EventHandler-3-1669745351 is a topology
> name+id, worker id 1d375d63-38e3-4b6d-8d9e-4db7dd9bc20e
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>