You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Aishwarya Soni <ai...@gmail.com> on 2023/06/01 19:20:23 UTC
Does storm 2.4.0 supports IPv6 address
Hi,
I have a storm 2.4.0 cluster with all the storm components such as Nimbus,
UI, supervisors, and drpc. Recently we moved to a new GCP Kubernetes
cluster where both traffic was coming from both IPv4 and IPv6 addresses.
Previously, we were on a hashicorp nomad cluster where only IPv4 was
supported.
Since the move to GCP Kubernetes (v1.23.x), I started to see supervisor is
unable to communicate with drpc on port 3773. This is the error snippet
from one of the topology logs inside a supervisor.
2023-06-01 19:02:10.136 o.a.s.d.DRPCSpout Thread-28-spout-executor[194,
194] [ERROR] Failed to fetch DRPC request from DRPC server
[search-storm-drpc-4.search-storm-drpc:3773]
org.apache.storm.thrift.transport.TTransportException: Socket is closed by
peer.
at
org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:130)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
~[storm-shaded-deps-2.4.0.jar:2.4.0]
at
org.apache.storm.generated.DistributedRPCInvocations$Client.recv_fetchRequest(DistributedRPCInvocations.java:114)
~[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.generated.DistributedRPCInvocations$Client.fetchRequest(DistributedRPCInvocations.java:101)
~[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:84)
~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:175)
[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:187)
[storm-client-2.4.0.jar:2.4.0]
at
org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:153)
[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.utils.Utils$1.run(Utils.java:396)
[storm-client-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
On further debugging, I see all storm-drpc ports (3772, 3773, 3774) are
listening to IPv6 addresses.
[image: image.png]
I tried adding the JVM flag *-Djava.net.preferIPv4Stack=true *mentioned
here https://storm.apache.org/releases/2.4.0/Troubleshooting.html but that
also did not help.
The weird thing is, telnet to the DRPC DNS names and ports work from all
pods, but still, I see socket errors in supervisor pods while talking to
DRPC.
Any inputs on any difficulties or workaround to run Storm 2.4.0 in IPv6
addresses or make it successfully run by making ports listen to IPv4
addresses?
Regards,
Ash
Re: Does storm 2.4.0 supports IPv6 address
Posted by Bipin Prasad via user <us...@storm.apache.org>.
Hello Aishwarya, Are you using a helm chart for this deploy and/or k8s deployment/service files that you can share? I don't believe Storm has been extensively tested in IPv6.
Thanks--Bipin
On Thursday, June 1, 2023 at 12:20:45 PM PDT, Aishwarya Soni <ai...@gmail.com> wrote:
Hi,
I have a storm 2.4.0 cluster with all the storm components such as Nimbus, UI, supervisors, and drpc. Recently we moved to a new GCP Kubernetes cluster where both traffic was coming from both IPv4 and IPv6 addresses. Previously, we were on a hashicorp nomad cluster where only IPv4 was supported.
Since the move to GCP Kubernetes (v1.23.x), I started to see supervisor is unable to communicate with drpc on port 3773. This is the error snippet from one of the topology logs inside a supervisor.
2023-06-01 19:02:10.136 o.a.s.d.DRPCSpout Thread-28-spout-executor[194, 194] [ERROR] Failed to fetch DRPC request from DRPC server [search-storm-drpc-4.search-storm-drpc:3773]
org.apache.storm.thrift.transport.TTransportException: Socket is closed by peer.
at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:130) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.generated.DistributedRPCInvocations$Client.recv_fetchRequest(DistributedRPCInvocations.java:114) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.generated.DistributedRPCInvocations$Client.fetchRequest(DistributedRPCInvocations.java:101) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:84) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:175) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:187) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:153) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.utils.Utils$1.run(Utils.java:396) [storm-client-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
On further debugging, I see all storm-drpc ports (3772, 3773, 3774) are listening to IPv6 addresses.
I tried adding the JVM flag -Djava.net.preferIPv4Stack=true mentioned here https://storm.apache.org/releases/2.4.0/Troubleshooting.html but that also did not help.
The weird thing is, telnet to the DRPC DNS names and ports work from all pods, but still, I see socket errors in supervisor pods while talking to DRPC.
Any inputs on any difficulties or workaround to run Storm 2.4.0 in IPv6 addresses or make it successfully run by making ports listen to IPv4 addresses?
Regards,
Ash
Re: Does storm 2.4.0 supports IPv6 address
Posted by Bipin Prasad <bi...@yahoo.com.INVALID>.
Hello Aishwarya, Are you using a helm chart for this deploy and/or k8s deployment/service files that you can share? I don't believe Storm has been extensively tested in IPv6.
Thanks--Bipin
On Thursday, June 1, 2023 at 12:20:45 PM PDT, Aishwarya Soni <ai...@gmail.com> wrote:
Hi,
I have a storm 2.4.0 cluster with all the storm components such as Nimbus, UI, supervisors, and drpc. Recently we moved to a new GCP Kubernetes cluster where both traffic was coming from both IPv4 and IPv6 addresses. Previously, we were on a hashicorp nomad cluster where only IPv4 was supported.
Since the move to GCP Kubernetes (v1.23.x), I started to see supervisor is unable to communicate with drpc on port 3773. This is the error snippet from one of the topology logs inside a supervisor.
2023-06-01 19:02:10.136 o.a.s.d.DRPCSpout Thread-28-spout-executor[194, 194] [ERROR] Failed to fetch DRPC request from DRPC server [search-storm-drpc-4.search-storm-drpc:3773]
org.apache.storm.thrift.transport.TTransportException: Socket is closed by peer.
at org.apache.storm.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:130) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:141) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TFramedTransport.read(TFramedTransport.java:109) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.transport.TTransport.readAll(TTransport.java:86) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[storm-shaded-deps-2.4.0.jar:2.4.0]
at org.apache.storm.generated.DistributedRPCInvocations$Client.recv_fetchRequest(DistributedRPCInvocations.java:114) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.generated.DistributedRPCInvocations$Client.fetchRequest(DistributedRPCInvocations.java:101) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCInvocationsClient.fetchRequest(DRPCInvocationsClient.java:84) ~[storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.drpc.DRPCSpout.nextTuple(DRPCSpout.java:175) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:187) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.executor.spout.SpoutExecutor$2.call(SpoutExecutor.java:153) [storm-client-2.4.0.jar:2.4.0]
at org.apache.storm.utils.Utils$1.run(Utils.java:396) [storm-client-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
On further debugging, I see all storm-drpc ports (3772, 3773, 3774) are listening to IPv6 addresses.
I tried adding the JVM flag -Djava.net.preferIPv4Stack=true mentioned here https://storm.apache.org/releases/2.4.0/Troubleshooting.html but that also did not help.
The weird thing is, telnet to the DRPC DNS names and ports work from all pods, but still, I see socket errors in supervisor pods while talking to DRPC.
Any inputs on any difficulties or workaround to run Storm 2.4.0 in IPv6 addresses or make it successfully run by making ports listen to IPv4 addresses?
Regards,
Ash