You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Kim, Peter" <Pe...@blackrock.com.INVALID> on 2020/10/21 18:55:49 UTC
Storm Socket Closed - Kerberos

Hello,

Recently, we’ve implemented storm 2.0.0 with Kerberos. Before integrating Kerberos, the logs were fine and no mention of socket output being closed but since we implemented Kerberos, we’re seeing the following in our logs :

2020-10-21 14:39:53.529 o.a.s.t.t.TIOStreamTransport refresh-connections-timer [WARN] Error closing output stream.
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116) ~[?:1.8.0_45]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_45]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_45]
        at org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.getLocalAssignment(WorkerState.java:647) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:367) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.WorkerState.refreshConnections(WorkerState.java:356) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker$$Lambda$61/1922191129.run(Unknown Source) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.t.t.TIOStreamTransport heartbeat-timer [WARN] Error closing output stream.
java.net.SocketException: Socket closed
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:116) ~[?:1.8.0_45]
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_45]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_45]
        at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_45]
        at org.apache.storm.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.thrift.transport.TSocket.close(TSocket.java:235) [storm-shaded-deps-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:106) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.<init>(SupervisorClient.java:45) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClientAs(SupervisorClient.java:73) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.utils.SupervisorClient.getConfiguredClient(SupervisorClient.java:61) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.heartbeatToMasterIfLocalbeatFail(Worker.java:429) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.doHeartBeat(Worker.java:344) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker.lambda$loadWorker$1(Worker.java:189) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.daemon.worker.Worker$$Lambda$30/435297131.run(Unknown Source) [storm-client-2.0.0.jar:2.0.0]
        at org.apache.storm.StormTimer$1.run(StormTimer.java:110) [storm-client-2.0.0.jar:2.0.0]
       at org.apache.storm.StormTimer$StormTimerTask.run(StormTimer.java:226) [storm-client-2.0.0.jar:2.0.0]
2020-10-21 14:43:37.827 o.a.s.d.w.Worker heartbeat-timer [WARN] Exception when send heartbeat to local supervisor

I’ve done some searching and tried adjusting the worker heartbeat frequency sec but that did more harm than good. I’ve followed the trace and the only configuration that I see that is being pulled in is the worker.heartbeat.freqs.secs set at 1.

Is this a known issue or has anyone come across this before? Any thoughts on what might be the issue and if it’s critical? Thanks for your help.

Best,
Peter


This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/compliance/email-disclaimers for further information.  Please refer to http://www.blackrock.com/corporate/compliance/privacy-policy for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.

© 2020 BlackRock, Inc. All rights reserved.