You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Mike Percy (JIRA)" <ji...@apache.org> on 2012/09/11 22:46:09 UTC
[jira] [Created] (AVRO-1156) Avro responder swallows thrown Errors
Mike Percy created AVRO-1156:
--------------------------------
Summary: Avro responder swallows thrown Errors
Key: AVRO-1156
URL: https://issues.apache.org/jira/browse/AVRO-1156
Project: Avro
Issue Type: Bug
Reporter: Mike Percy
Fix For: 1.7.2
The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453452#comment-13453452 ]
Doug Cutting commented on AVRO-1156:
------------------------------------
Also, if we want to go with the kill-the-JVM-on-Error approach, we should add some tests to make sure this happens. Such a change might be considered an incompatible change and might be pushed into 1.8.0 (or we might rename the next release 1.8.0 instead of 1.7.2).
What do servlet containers do with Errors in servlets? Do they close the connection, returning and logging nothing? Or do they risk zombification?
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453473#comment-13453473 ]
Mike Percy commented on AVRO-1156:
----------------------------------
To add a bit more information here, I'm seeing stack traces along the lines of the following:
WARN ipc.Responder: user error
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:93)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:158)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:143)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:299)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
... snip ...
Caused by: java.lang.OutOfMemoryError: Java heap space
and
WARN ipc.Responder: system error
org.apache.avro.AvroRuntimeException: Unknown datum type: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:574)
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:539)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:137)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:70)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
at org.apache.avro.ipc.specific.SpecificResponder.writeError(SpecificResponder.java:71)
at org.apache.avro.ipc.Responder.respond(Responder.java:167)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:158)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:143)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
... snip ...
at java.lang.Thread.run(Thread.java:662)
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453428#comment-13453428 ]
Philip Zeyliger commented on AVRO-1156:
---------------------------------------
Patch looks good to me. If there aren't any objections, I'll be happy to commit it.
Thanks, Mike!
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453455#comment-13453455 ]
Philip Zeyliger commented on AVRO-1156:
---------------------------------------
I misunderstood the patch, so withdraw my +1 for the time being.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453436#comment-13453436 ]
Doug Cutting commented on AVRO-1156:
------------------------------------
I think this change will just close the connection rather than kill the JVM. There are recoverable conditions that result in OutOfMemoryError, like 'new long[Integer.MAX_VALUE]', so killing the JVM may be overkill. The net effect of this change would be that clients, instead of receiving an exception that names the server-side error that occurred, will just see the connection closed. That seems like a step backwards, no?
If we really wanted to kill the JVM in a multi-threaded server I think we'd need to call Runtime.exit(). In some cases this might be better, since the alternative might leave a zombie server. But in other common cases, e.g., where a badly formatted request attempts a huge allocation, this will kill a healthy server. So I'm not very enthusiastic about this approach. The normal way to detect zombie servers is to add an isZombie() method to protocols and ping periodically to see whether the server is still responding sensibly.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1156) Avro responder swallows thrown Errors
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated AVRO-1156:
-----------------------------
Attachment: AVRO-1156-1.patch
Small patch so that if the Throwable is an Error, throw it as an Error.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453470#comment-13453470 ]
Brock Noland commented on AVRO-1156:
------------------------------------
I am +1 for throwing Error. Outside of some cases where you are actually allocating large amounts of memory or loading native libraries, catching Error(s) is dangerous. I have seen a large number of scenarios where this causes problems which are very hard to debug, as Mike mentions.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Scott Carey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453461#comment-13453461 ]
Scott Carey commented on AVRO-1156:
-----------------------------------
Usually if you do something like throw new Throwable() in a servlet container, the result is a HTTP 500 response.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453639#comment-13453639 ]
Mike Percy commented on AVRO-1156:
----------------------------------
Doug I agree with your last comment about improving the logging. The double OOM error is a hunch, I will try to find more evidence of it.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453477#comment-13453477 ]
Doug Cutting commented on AVRO-1156:
------------------------------------
If there's a memory leak through a static field then wrapping an exception for an OutOfMemoryError might cause another OutOfMemoryError. But if the OutOfMemoryError was caused by data only pointed to by the stack of the request then it should be collectable by the time we catch the exception and wrapping it shouldn't cause another OutOfMemoryError. Are you sure that's what's happening?
Try adding the extra logging there. If that gets something logged that wouldn't be otherwise then it's probably a good idea. But if it just results in double-logging of the error (since the container probably already logs errors) then it's probably not useful.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453462#comment-13453462 ]
Mike Percy commented on AVRO-1156:
----------------------------------
Thanks for the feedback guys! Re. the zombie detection, makes sense... there appear to be many benefits to doing it that way.
The issue I think I'm seeing is that an OOM error gets thrown while trying to service the request, so we wrap it in an Exception and try to handle it, so the Responder tries to allocate memory to serialize the error response and at that point another OOM error gets thrown by the JVM and we lose the original error.
I think trying to handle certain Errors in general is a tough proposition... maybe most Errors can be marshalled but OutOfMemoryError or VirtualMachineError should be special cased to just propagate? Or if we really want to try, maybe we should just log the Errors before wrapping them so at least we know if we are hitting a double-throw exception clobbering situation like I am suspecting.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1156) Avro responder swallows thrown Errors
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-1156:
-------------------------------
Fix Version/s: (was: 1.7.2)
Status: Open (was: Patch Available)
Let's hold off committing this just yet.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1156) Avro responder swallows thrown
Errors
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13453490#comment-13453490 ]
Doug Cutting commented on AVRO-1156:
------------------------------------
That logging doesn't look to me like a double OutOfMemoryError. The first message logs that the user code threw some exception (a wrapped OutOfMemoryError). The second message logs that the exception thrown by the user code is not one expected by the protocol.
This logging could probably be improved. If the Error wasn't caught by SpecificResponder then we'd probably want to change the "system error" catch clause in Responder.java to catch Throwable so the OutOfMemoryError is still logged and sent back to the user. In that case there'd only be a "system error" logged and not a "user error" as well.
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1156) Avro responder swallows thrown Errors
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated AVRO-1156:
-----------------------------
Assignee: Mike Percy
Status: Patch Available (was: Open)
> Avro responder swallows thrown Errors
> -------------------------------------
>
> Key: AVRO-1156
> URL: https://issues.apache.org/jira/browse/AVRO-1156
> Project: Avro
> Issue Type: Bug
> Reporter: Mike Percy
> Assignee: Mike Percy
> Fix For: 1.7.2
>
> Attachments: AVRO-1156-1.patch
>
>
> The Avro responder wraps caught Errors, such as OutOfMemoryErrors, in Exceptions and rethrows them. That's problematic because an Error should be allowed to crash the JVM, since it's often irrecoverable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira