Re address already bound issue

[JOLLIFFE Chloe <cj...@murex.com>]

Hello,


We have also encountered Address already bound issues as reported in
 https://lists.apache.org/thread/wbmtnjbmgb5h7hfn3q6dfpcozohk62c7 since upgrading to Storm 2.3.0.

In our case, the process that is running on the port is that of a "killed" Storm worker.
We have seen it most often with a supervisor with 1 worker slot (see log excerpts attached);
                              We have a successfully started topology on a worker
                              We start a second topology (waiting for a slot)
                              We attempt to kill the first topology
                                                            - The kill command is sent
                                                            - The topology begins to deactivate
                                                            - The supervisor releases the slot
                              Nimbus then assigns the waiting topology to the "free" slot

                              However, the first worker process is not dead/completely cleaned up yet.

                              This results in the second topology getting an "java.net.BindException: Address already in use" exception and shutting down.

I've not been able to determine the root cause of the issue, it appears it may be a race condition as after turning on some debug logs in the supervisor, I have not been able to reproduce.

Any help would be appreciated.

Thanks,
Chloe

*******************************
This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. Its content and any attachment hereto are strictly confidential and must not be disclosed to any unauthorized third party. If you are not the intended recipient, please delete this email and any attachment and notify us immediately. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.