You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by GitBox <gi...@apache.org> on 2020/06/26 15:23:19 UTC
[GitHub] [storm] RuiLi8080 opened a new pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
RuiLi8080 opened a new pull request #3296:
URL: https://github.com/apache/storm/pull/3296
## What is the purpose of the change
In some (rare) cases, we observed netty-server might receive messages other than a List<TaskMessage> from a wrong source, and it will throw ClassCastException and terminate the worker process.
## How was the change tested
Set netty server side to not using Sasl related ServerHandler while making client send sasl message, we observe the following message without worker getting killed
```
2020-06-26 15:10:41.086 o.a.s.m.n.Server Netty-server-localhost-6701-worker-1 [ERROR] Worker netty server receive message other than the expected class List<TaskMessage>
java.lang.ClassCastException: org.apache.storm.messaging.netty.ControlMessage cannot be cast to java.util.List
at org.apache.storm.messaging.netty.Server.received(Server.java:266) [storm-client-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) [storm-client-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a change in pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
agresch commented on a change in pull request #3296:
URL: https://github.com/apache/storm/pull/3296#discussion_r446253730
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage>", e);
Review comment:
Can we add remote? I assume that might add useful debug info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3296:
URL: https://github.com/apache/storm/pull/3296#discussion_r447039454
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage> from remote: {}", remote, e);
Review comment:
nit: `receive` -> `received` ;
And ad the log to indicate the message is ignored will be helpful
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage> from remote: {}", remote, e);
Review comment:
nit: `receive` -> `received` ;
And add the log to indicate the message is ignored will be helpful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm commented on a change in pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
Ethanlm commented on a change in pull request #3296:
URL: https://github.com/apache/storm/pull/3296#discussion_r447039454
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage> from remote: {}", remote, e);
Review comment:
nit: `receive` -> `received` ;
And add the log to indicate the message is ignored will be helpful
Also can you show the log after you added `remote`? Is it an Ip address? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] RuiLi8080 commented on a change in pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
RuiLi8080 commented on a change in pull request #3296:
URL: https://github.com/apache/storm/pull/3296#discussion_r446788932
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage>", e);
Review comment:
@agresch Remote added.
@bipinprasad Yes the exception message can be displayed just as the `ClassCastException` in the description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] Ethanlm merged pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
Ethanlm merged pull request #3296:
URL: https://github.com/apache/storm/pull/3296
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a change in pull request #3296: [STORM-3661] handle exception when netty server receive unexpected message
Posted by GitBox <gi...@apache.org>.
bipinprasad commented on a change in pull request #3296:
URL: https://github.com/apache/storm/pull/3296#discussion_r446290984
##########
File path: storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java
##########
@@ -260,7 +260,15 @@ public void channelActive(Channel c) {
@Override
public void received(Object message, String remote, Channel channel) throws InterruptedException {
- List<TaskMessage> msgs = (List<TaskMessage>) message;
+ List<TaskMessage> msgs;
+
+ try {
+ msgs = (List<TaskMessage>) message;
+ } catch (ClassCastException e) {
+ LOG.error("Worker netty server receive message other than the expected class List<TaskMessage>", e);
Review comment:
Will this log display the original exception message string that shows the Class of the offending message?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org