You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2009/02/27 12:31:13 UTC
[jira] Created: (HADOOP-5348) Create a stable wire format for
exceptions
Create a stable wire format for exceptions
------------------------------------------
Key: HADOOP-5348
URL: https://issues.apache.org/jira/browse/HADOOP-5348
Project: Hadoop Core
Issue Type: New Feature
Components: ipc
Affects Versions: 0.20.0, 0.21.0
Reporter: Steve Loughran
HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683427#action_12683427 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
Some sample exception classes, mostly soapfault related, that so similarish things
Axis 1.x AxisFault
http://svn.apache.org/viewvc/webservices/axis/trunk/java/src/org/apache/axis/AxisFault.java?view=markup
* lots of complexity related to XML fragments, parsing, namespaces -stuff to avoid.
* automatically adds hostname as one of the elements, very good for tracking down trouble in a cluster
* Security: the normal Axis servlet strips out the stack before sending the fault over the wire, unless the machine has a "development box" switch set.
Axis 2 AxisFault
http://svn.apache.org/viewvc/webservices/axis2/trunk/java/modules/kernel/src/org/apache/axis2/AxisFault.java?view=markup
* lots of SOAP1.2 complexity
* extracts the inner cause of a InvocationTargetException or UndeclaredThrowableException and discards the wrapper
* Security: has methods used to print the fault in an HTML page, methods that sanitize all strings by escaping angle brackets
Alpine faulting
http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/projects/alpine/prototype/M32/src/java/org/smartfrog/projects/alpine/faults/
* Lots of SOAP1.1/1.2 complexity
* has an interface, {{SoapFaultSource}}, that you can implement in any subclass of Throwable to say "I'll handle the fault to SOAP conversion".
* does some escaping of all received details, in case a malicious far end decides to send back malicious html as the fault name
* Can back-convert to an exception, but never tries to do the original (Because of this [www.hpl.hp.com/techreports/2005/HPL-2005-83.pdf] )
All of these suffer from the complexity of SOAP stacks, and the SOAP1.2 fault design in particular. what may be relevant is
# Hostnames are invaluable to retain, to add to new exceptions
# Being able to add name, value data is good, but full XML is far too much. String->string works well
# It's nice to have a client/server flag
# We may need to think of security, at least the use case of converting an exception into HTML to display on a report page.
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.20.0, 0.21.0
> Reporter: Steve Loughran
> Attachments: ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677329#action_12677329 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
This could also be used in the ping operations proposed in HADOOP-3628; these objects would be lighter weight than a list of exceptions if there was no need to create stack traces, but easy to marshall
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.20.0, 0.21.0
> Reporter: Steve Loughran
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677328#action_12677328 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
Proposed features
-something to take an exception and turn it (and any nested exceptions) into the wire format
-have a constructor that doesn't generate a stack trace, for lower cost construction of the exceptions
-retain things like classname and stack trace when creating the wire format
-ability for the far end to decode the exception into a class that clients with hadoop-core.jar will be guaranteed to have
-Utility code to push out into a human readable form in web interface
-possibly: a way to add a stable fault-code string that could be used for client side tests that was not as brittle as searching for text strings
Non-goals
-seamless conversion back into a throwable chain on the client side. We are not trying to do RMI here
-all the complexity of SOAP1.2 faults with actors, chained subfault-codes, etc. . We are not trying to do SOAP or JAX-WS here
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.20.0, 0.21.0
> Reporter: Steve Loughran
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12695376#action_12695376 ]
Hadoop QA commented on HADOOP-5348:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404535/hadoop-5348.patch
against trunk revision 761632.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/109/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/109/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/109/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/109/console
This message is automatically generated.
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683420#action_12683420 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
Needs of HADOOP-5201
# Ability to construct new StackTraceElement lists from stack traces (i,e. do not downconvert to text)
# ability to create a new exception from a deserialized exception, with toString(), stack trace &c recreating the details received
# Option of using error codes
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.20.0, 0.21.0
> Reporter: Steve Loughran
> Attachments: ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5348:
----------------------------------
Status: Open (was: Patch Available)
I think the plan of record is to standardize on Avro for Hadoop's wire format.
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-5348:
-----------------------------------
Fix Version/s: 0.21.0
Affects Version/s: (was: 0.20.0)
Status: Patch Available (was: Open)
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-5348:
-----------------------------------
Attachment: ThrowableWritable.java
First pass at this:
# you can create it from a chain of throwables, but you can't go the other way.
# Stack trace gets downgraded to string array
# No tests that conversion works or serialize/deserialize reliable
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.20.0, 0.21.0
> Reporter: Steve Loughran
> Attachments: ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5348) Create a ThrowableWritable for
serializing exceptions robustly
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-5348:
-----------------------------------
Summary: Create a ThrowableWritable for serializing exceptions robustly (was: Create a stable wire format for exceptions)
> Create a ThrowableWritable for serializing exceptions robustly
> --------------------------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706035#action_12706035 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
Maybe the term "wire format" is wrong; I should retitle this. Its about a stable Writable for exceptions that doesn't rely on the far end to have every exception in the Java chain. Rather than serialise the throwable and hope the far end has its classpath in perfect sync, right down to JDBC drivers, this turns the exception chain into a chain of ThrowableWritables, which can then be written and read.
We could add Avro support to this when the time comes.
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a ThrowableWritable for
serializing exceptions robustly
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706873#action_12706873 ]
Steve Loughran commented on HADOOP-5348:
----------------------------------------
It's certainly part of HADOOP-3628, and is on that branch, but my little HADOOP-5621 junit runner also uses it for reporting structured test failures -so it is more broadly useful than just service management.
> Create a ThrowableWritable for serializing exceptions robustly
> --------------------------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5348) Create a ThrowableWritable for
serializing exceptions robustly
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706688#action_12706688 ]
Chris Douglas commented on HADOOP-5348:
---------------------------------------
OK, I think I understand. Unfortunately, this issue proposes to add the type without any use cases in the framework. Absent context for this, it's hard to justify adding it to core on its own. If HADOOP-3628 can use it effectively (the ping functionality is a good idea), it would make more sense to add it as part of that work than to add it as an orphaned type.
> Create a ThrowableWritable for serializing exceptions robustly
> --------------------------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5348) Create a stable wire format for
exceptions
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran updated HADOOP-5348:
-----------------------------------
Attachment: hadoop-5348.patch
This is an updated patch with test case. It does not attempt to go from serialized form to exceptions, only from exceptions to wire form. The recipient is left to deal with it in this format.
You do not need to have an exception to create one of these objects -creating one with just a message is much lower cost, as stack tracing is expensive in a JVM.
> Create a stable wire format for exceptions
> ------------------------------------------
>
> Key: HADOOP-5348
> URL: https://issues.apache.org/jira/browse/HADOOP-5348
> Project: Hadoop Core
> Issue Type: New Feature
> Components: ipc
> Affects Versions: 0.21.0
> Reporter: Steve Loughran
> Fix For: 0.21.0
>
> Attachments: hadoop-5348.patch, ThrowableWritable.java
>
>
> HADOOP-5201 and other issues would benefit from a stable representation of exceptions, one that can be sent over the network, maybe pushed out to web UIs and which we can be 100% sure that the far end will be able to handle if they have the hadoop-core JAR on their classpath.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.