You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Raivis Strogonovs <ra...@udlive.io> on 2021/12/22 10:16:44 UTC
Re: Address already bound issue
Hello,
I was wondering if anybody could help me out with the issue I'm having for
the past couple months.
For some reason our storm instances are running fine for couple weeks, and
then suddenly it get's address bound issue, which can only be resolved by
restarting the respective machine. And it's not like it only happens one
machine, but randomly on any of them
We are still using 2.2.0 at the moment.
The instances are running in docker.
So far I haven't had luck at finding or figuring out, why would it happen...
See logs here
2021-12-22 10:03:39.057 o.a.s.m.n.Server main [INFO] Create Netty
Server Netty-server-localhost-6701, buffer_size: 5242880, maxWorkers:
1
2021-12-22 10:03:39.062 STDERR Thread-0 [INFO] WARNING: An illegal
reflective access operation has occurred
2021-12-22 10:03:39.075 STDERR Thread-0 [INFO] WARNING: Illegal
reflective access by com.esotericsoftware.kryo.util.UnsafeUtil
(file:/apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar) to constructor
java.nio.DirectByteBuffer(long,int,java.lang.Object)
2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Please
consider reporting this to the maintainers of
com.esotericsoftware.kryo.util.UnsafeUtil
2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Use
--illegal-access=warn to enable warnings of further illegal reflective
access operations
2021-12-22 10:03:39.082 STDERR Thread-0 [INFO] WARNING: All illegal
access operations will be denied in a future release
2021-12-22 10:03:39.144 o.a.s.u.Utils main [ERROR] Received error in
thread main.. terminating server...
java.lang.Error: java.security.PrivilegedActionException:
java.net.BindException: Address already in use
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:655)
~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:634)
~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1016)
~[storm-client-2.2.0.jar:2.2.0]
at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
at java.lang.Thread.dispatchUncaughtException(Unknown Source) [?:?]
Caused by: java.security.PrivilegedActionException
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Unknown Source) ~[?:?]
at org.apache.storm.daemon.worker.Worker.start(Worker.java:188)
~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.daemon.worker.Worker.main(Worker.java:153)
~[storm-client-2.2.0.jar:2.2.0]
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?]
at org.apache.storm.shade.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
~[storm-shaded-deps-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Any Ideas would be appreciated.
Thank You,
Raivis
--
Disclaimer:
The content of this email is confidential and intended for
the recipient specified in message only. It is strictly forbidden to share
any part of this message with any third party, without a written consent of
the sender. If you received this message by mistake, please reply to this
message and follow with its deletion, so that we can ensure such a mistake
does not occur in the future. The security of customers and suppliers as a
priority. Therefore, we have put efforts into ensuring that the message is
error and virus-free. Unfortunately, full security of the email cannot be
ensured as, despite our efforts, the data included in emails could be
infected, intercepted, or corrupted. Therefore, the recipient should check
the email for threats with proper software, as the sender does not accept
liability for any damage inflicted by viewing the content of this email.
No employee has the authority to conclude any binding contract without an
explicit written consent of their supervisor. Therefore, any will to enter
into an agreement must be confirmed by a second manager or director.
Any
quotation contained is sent to compare available offers and does not imply
entering into a legally binding contract.
Please do not print this email
unless it is necessary. Every unprinted email helps the environment.
The
views and opinions included in this email belong to their author and do not
necessarily mirror the views and opinions of the company. Our employees are
obliged not to make any defamatory clauses, infringe, or authorise
infringement of any legal right. Therefore, the company will not take any
liability for such statements included in emails. In case of any damages or
other liabilities arising, employees are fully responsible for the content
of their emails.
RE: [External] Re: Address already bound issue
Posted by Bipin Prasad <bi...@apache.org>.
This type of error means that some other process/thread is listening on that port. You can use lsof and kill that process.
On 2022/04/27 18:52:33 "Le, Binh T." wrote:
> We also just encountered this same issue as originally reported by Raivis. I'm not sure what is going on either. It just happened all of a sudden when a worker restarted.
>
> From: pwozniak <pw...@man.poznan.pl>
> Sent: Tuesday, February 15, 2022 8:28 AM
> To: user@storm.apache.org; raivis@udlive.io
> Subject: [External] Re: Address already bound issue
>
> This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
> ________________________________
>
>
> Hi,
>
> we encountered similar problem lately. In our case it was related with JMX port that we use for our worker. Quite strange in here is that it happened some time after worker startup (not during worker startup). Our Storm version is 2.3.0 (we lately upgraded it from 1.1.3).
>
>
> Any Ideas would be appreciated :)
>
> 2022-01-26 19:43:09.655 e.e.c.s.d.s.AbstractDpsBolt Thread-103-EDMObjectProcessorBolt-executor[17, 17] [INFO] Attempt number 2 to process this message. No cleaning done here.
>
>
>
> 2022-01-27 09:43:35.558 STDERR Thread-0 [INFO] Error: JMX connector server communication error: service:jmx:rmi://storm-supervisor2.novalocal:16700
>
> 2022-01-27 09:43:35.583 STDERR Thread-0 [INFO] jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 16700; nested exception is:
>
> 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed)
>
> 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:820)
>
> 2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
>
> 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
>
> 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
>
> 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] Caused by: java.rmi.server.ExportException: Port already in use: 16700; nested exception is:
>
> 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed)
>
> 2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
>
> java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
>
> 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
>
> java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
>
> 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
>
> java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
>
> 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
>
> java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
>
> 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
>
> java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
>
> 2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:203)
>
> 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
>
> java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153)
>
> 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
>
> java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138)
>
> 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
>
> java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:473)
>
> 2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
>
> jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
>
> 2022-01-27 09:43:35.588 STDERR Thread-0 [INFO] ... 3 more
>
>
>
> Regards,
>
> Pawel
>
>
> On 12/22/21 11:16 AM, Raivis Strogonovs wrote:
> Hello,
>
> I was wondering if anybody could help me out with the issue I'm having for the past couple months.
>
> For some reason our storm instances are running fine for couple weeks, and then suddenly it get's address bound issue, which can only be resolved by restarting the respective machine. And it's not like it only happens one machine, but randomly on any of them
>
> We are still using 2.2.0 at the moment.
> The instances are running in docker.
>
> So far I haven't had luck at finding or figuring out, why would it happen...
>
> See logs here
>
>
> 2021-12-22 10:03:39.057 o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6701, buffer_size: 5242880, maxWorkers: 1
>
> 2021-12-22 10:03:39.062 STDERR Thread-0 [INFO] WARNING: An illegal reflective access operation has occurred
>
> 2021-12-22 10:03:39.075 STDERR Thread-0 [INFO] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar<file://apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar>) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
>
> 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
>
> 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
>
> 2021-12-22 10:03:39.082 STDERR Thread-0 [INFO] WARNING: All illegal access operations will be denied in a future release
>
> 2021-12-22 10:03:39.144 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server...
>
> java.lang.Error: java.security.PrivilegedActionException: java.net.BindException: Address already in use
>
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:655) ~[storm-client-2.2.0.jar:2.2.0]
>
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:634) ~[storm-client-2.2.0.jar:2.2.0]
>
> at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1016) ~[storm-client-2.2.0.jar:2.2.0]
>
> at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
>
> at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
>
> at java.lang.Thread.dispatchUncaughtException(Unknown Source) [?:?]
>
> Caused by: java.security.PrivilegedActionException
>
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
>
> at javax.security.auth.Subject.doAs(Unknown Source) ~[?:?]
>
> at org.apache.storm.daemon.worker.Worker.start(Worker.java:188) ~[storm-client-2.2.0.jar:2.2.0]
>
> at org.apache.storm.daemon.worker.Worker.main(Worker.java:153) ~[storm-client-2.2.0.jar:2.2.0]
>
> Caused by: java.net.BindException: Address already in use
>
> at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
>
> at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
>
> at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
>
> at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?]
>
> at org.apache.storm.shade.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
>
> at java.lang.Thread.run(Unknown Source) ~[?:?]
>
> Any Ideas would be appreciated.
>
>
> Thank You,
> Raivis
>
>
> Disclaimer:
>
> The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
> The security of customers and suppliers as a priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.
> No employee has the authority to conclude any binding contract without an explicit written consent of their supervisor. Therefore, any will to enter into an agreement must be confirmed by a second manager or director.
> Any quotation contained is sent to compare available offers and does not imply entering into a legally binding contract.
> Please do not print this email unless it is necessary. Every unprinted email helps the environment.
> The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorise infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.
>
> ________________________________
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
> ______________________________________________________________________________________
>
> www.accenture.com
>
RE: [External] Re: Address already bound issue
Posted by "Le, Binh T." <bi...@accenture.com>.
We also just encountered this same issue as originally reported by Raivis. I'm not sure what is going on either. It just happened all of a sudden when a worker restarted.
From: pwozniak <pw...@man.poznan.pl>
Sent: Tuesday, February 15, 2022 8:28 AM
To: user@storm.apache.org; raivis@udlive.io
Subject: [External] Re: Address already bound issue
This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with links and attachments.
________________________________
Hi,
we encountered similar problem lately. In our case it was related with JMX port that we use for our worker. Quite strange in here is that it happened some time after worker startup (not during worker startup). Our Storm version is 2.3.0 (we lately upgraded it from 1.1.3).
Any Ideas would be appreciated :)
2022-01-26 19:43:09.655 e.e.c.s.d.s.AbstractDpsBolt Thread-103-EDMObjectProcessorBolt-executor[17, 17] [INFO] Attempt number 2 to process this message. No cleaning done here.
2022-01-27 09:43:35.558 STDERR Thread-0 [INFO] Error: JMX connector server communication error: service:jmx:rmi://storm-supervisor2.novalocal:16700
2022-01-27 09:43:35.583 STDERR Thread-0 [INFO] jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 16700; nested exception is:
2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed)
2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:820)
2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] Caused by: java.rmi.server.ExportException: Port already in use: 16700; nested exception is:
2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] java.net.BindException: Address already in use (Bind failed)
2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:203)
2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153)
2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138)
2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:473)
2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
2022-01-27 09:43:35.588 STDERR Thread-0 [INFO] ... 3 more
Regards,
Pawel
On 12/22/21 11:16 AM, Raivis Strogonovs wrote:
Hello,
I was wondering if anybody could help me out with the issue I'm having for the past couple months.
For some reason our storm instances are running fine for couple weeks, and then suddenly it get's address bound issue, which can only be resolved by restarting the respective machine. And it's not like it only happens one machine, but randomly on any of them
We are still using 2.2.0 at the moment.
The instances are running in docker.
So far I haven't had luck at finding or figuring out, why would it happen...
See logs here
2021-12-22 10:03:39.057 o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6701, buffer_size: 5242880, maxWorkers: 1
2021-12-22 10:03:39.062 STDERR Thread-0 [INFO] WARNING: An illegal reflective access operation has occurred
2021-12-22 10:03:39.075 STDERR Thread-0 [INFO] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar<file://apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar>) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Please consider reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2021-12-22 10:03:39.082 STDERR Thread-0 [INFO] WARNING: All illegal access operations will be denied in a future release
2021-12-22 10:03:39.144 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server...
java.lang.Error: java.security.PrivilegedActionException: java.net.BindException: Address already in use
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:655) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:634) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1016) ~[storm-client-2.2.0.jar:2.2.0]
at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
at java.lang.Thread.dispatchUncaughtException(Unknown Source) [?:?]
Caused by: java.security.PrivilegedActionException
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Unknown Source) ~[?:?]
at org.apache.storm.daemon.worker.Worker.start(Worker.java:188) ~[storm-client-2.2.0.jar:2.2.0]
at org.apache.storm.daemon.worker.Worker.main(Worker.java:153) ~[storm-client-2.2.0.jar:2.2.0]
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?]
at org.apache.storm.shade.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Any Ideas would be appreciated.
Thank You,
Raivis
Disclaimer:
The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
The security of customers and suppliers as a priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.
No employee has the authority to conclude any binding contract without an explicit written consent of their supervisor. Therefore, any will to enter into an agreement must be confirmed by a second manager or director.
Any quotation contained is sent to compare available offers and does not imply entering into a legally binding contract.
Please do not print this email unless it is necessary. Every unprinted email helps the environment.
The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorise infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. Your privacy is important to us. Accenture uses your personal data only in compliance with data protection laws. For further information on how Accenture processes your personal data, please see our privacy statement at https://www.accenture.com/us-en/privacy-policy.
______________________________________________________________________________________
www.accenture.com
Re: Address already bound issue
Posted by pwozniak <pw...@man.poznan.pl>.
Hi,
we encountered similar problem lately. In our case it was related with
JMX port that we use for our worker. Quite strange in here is that it
happened some time after worker startup (not during worker startup). Our
Storm version is 2.3.0 (we lately upgraded it from 1.1.3).
Any Ideas would be appreciated :)
|2022-01-26 19:43:09.655 e.e.c.s.d.s.AbstractDpsBolt
Thread-103-EDMObjectProcessorBolt-executor[17, 17] [INFO] Attempt number
2 to process this message. No cleaning done here. ||||2022-01-27
09:43:35.558 STDERR Thread-0 [INFO] Error: JMX connector server
communication error: service:jmx:rmi://storm-supervisor2.novalocal:16700
||2022-01-27 09:43:35.583 STDERR Thread-0 [INFO]
jdk.internal.agent.AgentConfigurationError:
java.rmi.server.ExportException: Port already in use: 16700; nested
exception is: ||2022-01-27 09:43:35.584 STDERR Thread-0 [INFO]
java.net.BindException: Address already in use (Bind failed)
||2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:820)
||2022-01-27 09:43:35.584 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:479)
||2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
||2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
||2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] Caused by:
java.rmi.server.ExportException: Port already in use: 16700; nested
exception is: ||2022-01-27 09:43:35.585 STDERR Thread-0 [INFO]
java.net.BindException: Address already in use (Bind failed)
||2022-01-27 09:43:35.585 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
||2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:243)
||2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:412)
||2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
||2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
java.rmi/sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:234)
||2022-01-27 09:43:35.586 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:203)
||2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:153)
||2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:138)
||2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:473)
||2022-01-27 09:43:35.587 STDERR Thread-0 [INFO] at
jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:816)
||2022-01-27 09:43:35.588 STDERR Thread-0 [INFO] ... 3 more|
Regards,
Pawel
On 12/22/21 11:16 AM, Raivis Strogonovs wrote:
> Hello,
>
> I was wondering if anybody could help me out with the issue I'm having
> for the past couple months.
>
> For some reason our storm instances are running fine for couple weeks,
> and then suddenly it get's address bound issue, which can only be
> resolved by restarting the respective machine. And it's not like it
> only happens one machine, but randomly on any of them
>
> We are still using 2.2.0 at the moment.
> The instances are running in docker.
>
> So far I haven't had luck at finding or figuring out, why would it
> happen...
>
> See logs here
>
> 2021-12-22 10:03:39.057 o.a.s.m.n.Server main [INFO] Create Netty Server Netty-server-localhost-6701, buffer_size: 5242880, maxWorkers: 1
> 2021-12-22 10:03:39.062 STDERR Thread-0 [INFO] WARNING: An illegal reflective access operation has occurred
> 2021-12-22 10:03:39.075 STDERR Thread-0 [INFO] WARNING: Illegal reflective access by com.esotericsoftware.kryo.util.UnsafeUtil (file:/apache-storm-2.2.0/lib-worker/kryo-3.0.3.jar) to constructor java.nio.DirectByteBuffer(long,int,java.lang.Object)
> 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Please consider
reporting this to the maintainers of com.esotericsoftware.kryo.util.UnsafeUtil
> 2021-12-22 10:03:39.078 STDERR Thread-0 [INFO] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
> 2021-12-22 10:03:39.082 STDERR Thread-0 [INFO] WARNING: All illegal access operations will be denied in a future release
> 2021-12-22 10:03:39.144 o.a.s.u.Utils main [ERROR] Received error in thread main.. terminating server...
> java.lang.Error: java.security.PrivilegedActionException: java.net.BindException: Address already in use
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:655) ~[storm-client-2.2.0.jar:2.2.0]
> at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:634) ~[storm-client-2.2.0.jar:2.2.0]
> at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:1016) ~[storm-client-2.2.0.jar:2.2.0]
> at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
> at java.lang.ThreadGroup.uncaughtException(Unknown Source) [?:?]
> at java.lang.Thread.dispatchUncaughtException(Unknown Source) [?:?]
> Caused by: java.security.PrivilegedActionException
> at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
> at javax.security.auth.Subject.doAs(Unknown Source) ~[?:?]
> at org.apache.storm.daemon.worker.Worker.start(Worker.java:188) ~[storm-client-2.2.0.jar:2.2.0]
> at org.apache.storm.daemon.worker.Worker.main(Worker.java:153) ~[storm-client-2.2.0.jar:2.2.0]
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
> at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
> at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
> at sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) ~[?:?]
> at org.apache.storm.shade.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) ~[storm-shaded-deps-2.2.0.jar:2.2.0]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
>
> Any Ideas would be appreciated.
>
>
> Thank You,
> Raivis
>
> Disclaimer:
>
> The content of this email is confidential and intended for the
> recipient specified in message only. It is strictly forbidden to share
> any part of this message with any third party, without a written
> consent of the sender. If you received this message by mistake, please
> reply to this message and follow with its deletion, so that we can
> ensure such a mistake does not occur in the future.
> The security of customers and suppliers as a priority. Therefore, we
> have put efforts into ensuring that the message is error and
> virus-free. Unfortunately, full security of the email cannot be
> ensured as, despite our efforts, the data included in emails could be
> infected, intercepted, or corrupted. Therefore, the recipient should
> check the email for threats with proper software, as the sender does
> not accept liability for any damage inflicted by viewing the content
> of this email.
> No employee has the authority to conclude any binding contract without
> an explicit written consent of their supervisor. Therefore, any will
> to enter into an agreement must be confirmed by a second manager or
> director.
> Any quotation contained is sent to compare available offers and does
> not imply entering into a legally binding contract.
> Please do not print this email unless it is necessary. Every unprinted
> email helps the environment.
> The views and opinions included in this email belong to their author
> and do not necessarily mirror the views and opinions of the company.
> Our employees are obliged not to make any defamatory clauses,
> infringe, or authorise infringement of any legal right. Therefore, the
> company will not take any liability for such statements included in
> emails. In case of any damages or other liabilities arising, employees
> are fully responsible for the content of their emails.