You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Patrick Hunt (JIRA)" <ji...@apache.org> on 2012/11/09 23:45:12 UTC
[jira] [Created] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Patrick Hunt created MRUNIT-158:
-----------------------------------
Summary: withOutput doesn't generate useful information with MapWritable on error
Key: MRUNIT-158
URL: https://issues.apache.org/jira/browse/MRUNIT-158
Project: MRUnit
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Patrick Hunt
Priority: Critical
Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497849#comment-13497849 ]
Bertrand Dechoux commented on MRUNIT-158:
-----------------------------------------
Like I said, for MapWritable the toString()/equals()/hashcode() methods were not overrided in the implementation.
This implies that
1) if the toString() outputs are different, then java will consider the MapWritable instances as different. Only by reading the output of your test, you indeed know why it is failing. (e1 =! 2059)
2) even if two MapWritable instances have exactly the same content (which implies that they should be equals according the java Map contract), they won't in practice actually be equal
A solution for MRunit could be to provide our own equal implementation that should be used only when the class of the key/value is an instance of MapWritable but that would means MRUnit behaviour will differ from the standard Hadoop behaviour. Introducing such potential bugs in a testing library is dodgy.
MRunit could also provide a custom toString() method using the same mechanism but that would render the test less explicit when comparing two MapWritable instances with the same content. The toString() output will be the same but that instances won't be equals. That would also be quite confusing.
Of course, MRunit is an apache project, you are welcome to correct me. And if you have a solution, you are also welcome to provide it or at least outline it.
For me, from the user point of view, the solution is to not use MapWritable. Sure, they are handy and that would be great if they were actually respecting the java Map contract but that's not the case as of now. And from a performance point of view, maps are not ideally because you are transporting a (untyped) schema for each instance.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494410#comment-13494410 ]
Bertrand Dechoux commented on MRUNIT-158:
-----------------------------------------
I wouldn't put that as critical. I am not even sure that doing anything for that subject would be a good thing.
Of course, a user friendly support of map would be nice but the underlying issue is with Haddop implementation of MapWritable. As of 1.03, there is no (overriding) implementation of toString(), equals() and hascode() methods. (see HADOOP-7153)
MRunit could detect the type being a MapWritable and in that case use its own definition of toString(), equals() and hashcode() but that would be a feature and not a bug fixe and I would not really be favorable to it.
Maps are indeed nice to transfer 'dynamic data' but it implies that each object carries its own typeless schema so that's not very efficient. You may want to look at tuples instead (and at library such as Cascading.)
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Critical
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MRUNIT-158) withOutput doesn't generate
useful information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494410#comment-13494410 ]
Bertrand Dechoux edited comment on MRUNIT-158 at 11/9/12 11:15 PM:
-------------------------------------------------------------------
I wouldn't put that as critical. I am not even sure that doing anything for that subject would be a good thing.
Of course, a user friendly support of map would be nice but the underlying issue is with Haddop implementation of MapWritable. As of 1.03, there is no (overriding) implementation of toString(), equals() and hascode() methods. (see HADOOP-7153)
MRunit could detect the type being a MapWritable and in that case use its own definition of toString(), equals() and hashcode() but that would be a feature and not a bug fixe and I would not really be favorable to it.
Maps are indeed nice to transfer 'dynamic data' but it implies that each object carries its own schema so that's not very efficient. You may want to look at tuples instead (and at library such as Cascading.)
was (Author: bdechoux):
I wouldn't put that as critical. I am not even sure that doing anything for that subject would be a good thing.
Of course, a user friendly support of map would be nice but the underlying issue is with Haddop implementation of MapWritable. As of 1.03, there is no (overriding) implementation of toString(), equals() and hascode() methods. (see HADOOP-7153)
MRunit could detect the type being a MapWritable and in that case use its own definition of toString(), equals() and hashcode() but that would be a feature and not a bug fixe and I would not really be favorable to it.
Maps are indeed nice to transfer 'dynamic data' but it implies that each object carries its own typeless schema so that's not very efficient. You may want to look at tuples instead (and at library such as Cascading.)
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Beech resolved MRUNIT-158.
-------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
I've just committed a patch to add log warnings if the user's key and value types do not override equals, hashCode or toString(). Please re-open if there's anything else you feel should go in to help resolve this.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Assignee: Dave Beech
> Priority: Minor
> Fix For: 1.0.0
>
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "posa Wu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500917#comment-13500917 ]
posa Wu commented on MRUNIT-158:
--------------------------------
I can not agree more.I was stuck by a problem and I have to debug the mrunit source to fix it, this is not a good idea..
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MRUNIT-158) withOutput doesn't generate
useful information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497849#comment-13497849 ]
Bertrand Dechoux edited comment on MRUNIT-158 at 11/15/12 8:50 AM:
-------------------------------------------------------------------
Like I said, for MapWritable the toString()/equals()/hashcode() methods were not overrided in the implementation.
This implies that
1) if the toString() outputs are different, then java will consider the MapWritable instances as different. Only by reading the output of your test, you indeed know why it is failing. (e1 =! 2059)
2) even if two MapWritable instances have exactly the same content (which implies that they should be equals according the java Map contract), they won't in practice actually be equal
A solution for MRunit could be to provide our own equal implementation that should be used only when the class of the key/value is an instance of MapWritable but that would means MRUnit behaviour will differ from the standard Hadoop behaviour. Introducing such potential bugs in a testing library is dodgy.
MRunit could also provide a custom toString() method using the same mechanism but that would render the test less explicit when comparing two MapWritable instances with the same content. The toString() output will be the same but that instances won't be equals. That would also be quite confusing.
Of course, MRunit is an apache project, you are welcome to correct me. And if you have a solution, you are also welcome to provide it or at least outline it.
For me, from the user point of view, the solution is to not use MapWritable. Sure, they are handy and that would be great if they were actually respecting the java Map contract but that's not the case as of now. And from a performance point of view, maps are not ideal because you are transporting a (untyped) schema for each instance.
was (Author: bdechoux):
Like I said, for MapWritable the toString()/equals()/hashcode() methods were not overrided in the implementation.
This implies that
1) if the toString() outputs are different, then java will consider the MapWritable instances as different. Only by reading the output of your test, you indeed know why it is failing. (e1 =! 2059)
2) even if two MapWritable instances have exactly the same content (which implies that they should be equals according the java Map contract), they won't in practice actually be equal
A solution for MRunit could be to provide our own equal implementation that should be used only when the class of the key/value is an instance of MapWritable but that would means MRUnit behaviour will differ from the standard Hadoop behaviour. Introducing such potential bugs in a testing library is dodgy.
MRunit could also provide a custom toString() method using the same mechanism but that would render the test less explicit when comparing two MapWritable instances with the same content. The toString() output will be the same but that instances won't be equals. That would also be quite confusing.
Of course, MRunit is an apache project, you are welcome to correct me. And if you have a solution, you are also welcome to provide it or at least outline it.
For me, from the user point of view, the solution is to not use MapWritable. Sure, they are handy and that would be great if they were actually respecting the java Map contract but that's not the case as of now. And from a performance point of view, maps are not ideally because you are transporting a (untyped) schema for each instance.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497880#comment-13497880 ]
Dave Beech commented on MRUNIT-158:
-----------------------------------
The lack of a toString override will always produce a less-than-ideal error message, but I suppose we could do a check to see if the Writable types being used override hashCode and equals, and print a loud warning if they don't.
Something like:
Class<? extends Writable> clazz = outputKey.getClass();
if (clazz.getMethod("equals", Object.class).getDeclaringClass() != clazz) {
LOG.warn("Equals method is not being overridden - tests may fail");
}
... repeat for value types, hashCode method, etc etc ...
We could do this either at the time expected output is added, or at the top of the validate method before checking outputs.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bertrand Dechoux updated MRUNIT-158:
------------------------------------
Priority: Minor (was: Critical)
Issue Type: Improvement (was: Bug)
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501114#comment-13501114 ]
Dave Beech commented on MRUNIT-158:
-----------------------------------
Hi Patrick. MRUnit already has this feature. If you run your test using the run() method rather than runTest(), you are provided with all the output pairs in a list which you can iterate through and do whatever you like with.
With regard to useful toString information for MapWritable, this should absolutely be a JIRA on Hadoop rather than here.
Posa Wu - could you provide more information about your problem? Do you need to open a new issue on JIRA for this too?
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501985#comment-13501985 ]
Bertrand Dechoux commented on MRUNIT-158:
-----------------------------------------
After reading the diff again from HADOOP-7153, it does not seem like the SortedMapWritable has been fixed in the same way. So there is definitely work to do within the Hadoop project.
Patrick : you can create an HADOOP issue (or reopen the old one). We will add the reference inside the MRUNIT jira for tracking it back. It is a recurring issue as it can been seen from the jira description.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497938#comment-13497938 ]
Bertrand Dechoux commented on MRUNIT-158:
-----------------------------------------
I really like that proposal. It will indeed help the user and it does not introduce behaviour that could be regretted later on.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500889#comment-13500889 ]
Patrick Hunt commented on MRUNIT-158:
-------------------------------------
Don't get me wrong here, I see the issues you are mentioning. However my context was that of a user of mrunit. They (me in this case) are trying to determine why a test was failing. As far as I could see there was no easy way to do that. Saying not to use MapWritable is not acceptable, there may be a perfectly valid reason for the user to do so, mrunit should support this case.
Looks like part of this was addressed in 0.23? https://issues.apache.org/jira/browse/HADOOP-7153 (I was running Hadoop code after this fix went in, so I had the equals/hashcode fixed, so really just the display of why the test failed was missing). Perhaps a jira should be created for Hadoop to add toString method to MapWritable?
Looking for this problem and printing a warning seems reasonable, however it would be nicer if mrunit could also provide some help in determining why the test is failing. Perhaps there is a way to simply provide the "output" to the user, so that they could extract additional information from that result? For example in my case if I could get access to the "output" I could trying iterating the entries and printing those...
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501320#comment-13501320 ]
Patrick Hunt commented on MRUNIT-158:
-------------------------------------
bq. If you run your test using the run() method rather than runTest(), you are provided with all the output pairs in a list which you can iterate through and do whatever you like with.
cool, this was not obvious to me. Perhaps the suggestion from Dave Beech could be combined with this? i.e. when the warn message is output it could include a reference to using the run method to get more detail and work around the limitation of MapWritable (et. al.)
I will enter a jira against Hadoop re adding the toString.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507757#comment-13507757 ]
Hudson commented on MRUNIT-158:
-------------------------------
Integrated in mrunit-trunk #490 (See [https://builds.apache.org/job/mrunit-trunk/490/])
MRUNIT-158: withOutput doesn't generate useful information with MapWritable on error (Revision 2f9b6f787f9ceeb18509f5d984fe24f8db29cf42)
Result = SUCCESS
dbeech : https://git-wip-us.apache.org/repos/asf?p=mrunit.git&a=commit&h=2f9b6f787f9ceeb18509f5d984fe24f8db29cf42
Files :
* src/main/java/org/apache/hadoop/mrunit/TestDriver.java
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Assignee: Dave Beech
> Priority: Minor
> Fix For: 1.0.0
>
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497954#comment-13497954 ]
Dave Beech commented on MRUNIT-158:
-----------------------------------
Great, thanks.
Patrick - would this change be enough to resolve the issue in your eyes? You wouldn't get any more detail about the contents of your MapWritable instances because of the lack of toString, but you would get an idea as to why the tests fail.
If this sounds OK I'll create a patch later.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497823#comment-13497823 ]
Patrick Hunt commented on MRUNIT-158:
-------------------------------------
Perhaps I missed it, is there another way to easily determine why my MRUnit test was failing? (other than say running a debugger)
I listed it as a bug/critical due to the fact that I couldn't actually figure out why my test was failing. Isn't that reasonable to expect to be able to do here?
I eventually subclassed MapWritable and added my own toString, that allowed me to see why the test was failing, however it assumes the user will be able to modify the source (which they won't in all cases if including some third party code).
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Beech reassigned MRUNIT-158:
---------------------------------
Assignee: Dave Beech
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Assignee: Dave Beech
> Priority: Minor
> Fix For: 1.0.0
>
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497860#comment-13497860 ]
Bertrand Dechoux commented on MRUNIT-158:
-----------------------------------------
If the user don't want or can't change the implementations of the key/value classes, a solution is to use the run() method of the drivers and check by themselves that the outputs are what they should be.
List<Pair<?, ?>> outputs = driver.run();
But we agree that's quite a verbose solution for something that should be trivial.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-158) withOutput doesn't generate useful
information with MapWritable on error
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502157#comment-13502157 ]
Patrick Hunt commented on MRUNIT-158:
-------------------------------------
I will create/reopen the hadoop jira.
Seems like the previous comment about printing a warning message is a good one - anyone can create a writable and if there is an issue with it not being compatible with mrunit having the user get notified would be useful. Providing insight into how to get around the issue (ie using run()) would be even more useful.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MRUNIT-158) withOutput doesn't generate
useful information with MapWritable on error
Posted by "Bertrand Dechoux (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497938#comment-13497938 ]
Bertrand Dechoux edited comment on MRUNIT-158 at 11/15/12 11:07 AM:
--------------------------------------------------------------------
I really like that proposal. It will indeed help the user and it does not introduce behaviour that could be regretted later on.
(And its arguably more constructive than my answers.)
was (Author: bdechoux):
I really like that proposal. It will indeed help the user and it does not introduce behaviour that could be regretted later on.
> withOutput doesn't generate useful information with MapWritable on error
> ------------------------------------------------------------------------
>
> Key: MRUNIT-158
> URL: https://issues.apache.org/jira/browse/MRUNIT-158
> Project: MRUnit
> Issue Type: Improvement
> Affects Versions: 0.9.0
> Reporter: Patrick Hunt
> Priority: Minor
>
> Using withOutput generates the following, notice that the hashcode is printed rather than some useful detail as to the content of the maps:
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Missing expected output (null-0, org.apache.hadoop.io.MapWritable@e1) at position 0.
> Nov 9, 2012 2:40:01 PM org.apache.hadoop.mrunit.TestDriver logError
> SEVERE: Received unexpected output (null-0, org.apache.hadoop.io.MapWritable@2059) at position 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira