You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Jim Donofrio (JIRA)" <ji...@apache.org> on 2012/04/23 00:43:32 UTC
[jira] [Created] (MRUNIT-105) java serialization tests fail to set
group comparator
Jim Donofrio created MRUNIT-105:
-----------------------------------
Summary: java serialization tests fail to set group comparator
Key: MRUNIT-105
URL: https://issues.apache.org/jira/browse/MRUNIT-105
Project: MRUnit
Issue Type: Bug
Affects Versions: 0.9.0
Reporter: Jim Donofrio
Assignee: Jim Donofrio
Fix For: 1.0.0
>From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.WritableComparable
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
at java.util.TreeMap.compare(TreeMap.java:1188)
at java.util.TreeMap.put(TreeMap.java:531)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
at
org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
public int compare(Object a, Object b) {
return compare((WritableComparable)a, (WritableComparable)b);
}
Java 7:
Entry<K,V> t = root;
if (t == null) {
compare(key, key); // type (and possibly null) check
root = new Entry<>(key, value, null);
size = 1;
modCount++;
return null;
}
Java 6:
Entry<K,V> t = root;
if (t == null) {
// TBD:
// 5045147: (coll) Adding null to an empty TreeSet should
// throw NullPointerException
//
// compare(key, key); // type check
root = new Entry<K,V>(key, value, null);
size = 1;
modCount++;
return null;
}
Oops this is a bug in test case not the actual code, good catch. This
problem is not unique to Java 7. The problem only appears in Java 7 because
my java serialization tests only use 1 input value. Java 6's put method in
TreeMap does not call compare when adding to an empty map while Java 7's put
method does call compare when adding to an empty map.
This can be fixed in the test case by setting a
OutputValueGroupingComparator in the conf for Integer. Users that use java
or other non Writable serializations would know they have to set the
relevant comparators.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-105) testJavaSerialization methods fail
in Java 7 because they do not set a group comparator
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio resolved MRUNIT-105.
---------------------------------
Resolution: Fixed
> testJavaSerialization methods fail in Java 7 because they do not set a group comparator
> ---------------------------------------------------------------------------------------
>
> Key: MRUNIT-105
> URL: https://issues.apache.org/jira/browse/MRUNIT-105
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.WritableComparable
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
> at java.util.TreeMap.compare(TreeMap.java:1188)
> at java.util.TreeMap.put(TreeMap.java:531)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
> at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
> at
> org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
> public int compare(Object a, Object b) {
> return compare((WritableComparable)a, (WritableComparable)b);
> }
> Java 7:
> Entry<K,V> t = root;
> if (t == null) {
> compare(key, key); // type (and possibly null) check
> root = new Entry<>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Java 6:
> Entry<K,V> t = root;
> if (t == null) {
> // TBD:
> // 5045147: (coll) Adding null to an empty TreeSet should
> // throw NullPointerException
> //
> // compare(key, key); // type check
> root = new Entry<K,V>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Oops this is a bug in test case not the actual code, good catch. This
> problem is not unique to Java 7. The problem only appears in Java 7 because
> the java serialization tests only use 1 input value. Java 6's put method in
> TreeMap does not call compare when adding to an empty map while Java 7's put
> method does call compare when adding to an empty map.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-105) java serialization tests fail because
they do not set a group comparator
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-105:
--------------------------------
Description:
>From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.WritableComparable
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
at java.util.TreeMap.compare(TreeMap.java:1188)
at java.util.TreeMap.put(TreeMap.java:531)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
at
org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
public int compare(Object a, Object b) {
return compare((WritableComparable)a, (WritableComparable)b);
}
Java 7:
Entry<K,V> t = root;
if (t == null) {
compare(key, key); // type (and possibly null) check
root = new Entry<>(key, value, null);
size = 1;
modCount++;
return null;
}
Java 6:
Entry<K,V> t = root;
if (t == null) {
// TBD:
// 5045147: (coll) Adding null to an empty TreeSet should
// throw NullPointerException
//
// compare(key, key); // type check
root = new Entry<K,V>(key, value, null);
size = 1;
modCount++;
return null;
}
Oops this is a bug in test case not the actual code, good catch. This
problem is not unique to Java 7. The problem only appears in Java 7 because
the java serialization tests only use 1 input value. Java 6's put method in
TreeMap does not call compare when adding to an empty map while Java 7's put
method does call compare when adding to an empty map.
was:
>From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
java.lang.ClassCastException: java.lang.Integer cannot be cast to
org.apache.hadoop.io.WritableComparable
at
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
at java.util.TreeMap.compare(TreeMap.java:1188)
at java.util.TreeMap.put(TreeMap.java:531)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
at
org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
at
org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
public int compare(Object a, Object b) {
return compare((WritableComparable)a, (WritableComparable)b);
}
Java 7:
Entry<K,V> t = root;
if (t == null) {
compare(key, key); // type (and possibly null) check
root = new Entry<>(key, value, null);
size = 1;
modCount++;
return null;
}
Java 6:
Entry<K,V> t = root;
if (t == null) {
// TBD:
// 5045147: (coll) Adding null to an empty TreeSet should
// throw NullPointerException
//
// compare(key, key); // type check
root = new Entry<K,V>(key, value, null);
size = 1;
modCount++;
return null;
}
Oops this is a bug in test case not the actual code, good catch. This
problem is not unique to Java 7. The problem only appears in Java 7 because
my java serialization tests only use 1 input value. Java 6's put method in
TreeMap does not call compare when adding to an empty map while Java 7's put
method does call compare when adding to an empty map.
This can be fixed in the test case by setting a
OutputValueGroupingComparator in the conf for Integer. Users that use java
or other non Writable serializations would know they have to set the
relevant comparators.
Summary: java serialization tests fail because they do not set a group comparator (was: java serialization tests fail to set group comparator)
> java serialization tests fail because they do not set a group comparator
> ------------------------------------------------------------------------
>
> Key: MRUNIT-105
> URL: https://issues.apache.org/jira/browse/MRUNIT-105
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.WritableComparable
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
> at java.util.TreeMap.compare(TreeMap.java:1188)
> at java.util.TreeMap.put(TreeMap.java:531)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
> at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
> at
> org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
> public int compare(Object a, Object b) {
> return compare((WritableComparable)a, (WritableComparable)b);
> }
> Java 7:
> Entry<K,V> t = root;
> if (t == null) {
> compare(key, key); // type (and possibly null) check
> root = new Entry<>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Java 6:
> Entry<K,V> t = root;
> if (t == null) {
> // TBD:
> // 5045147: (coll) Adding null to an empty TreeSet should
> // throw NullPointerException
> //
> // compare(key, key); // type check
> root = new Entry<K,V>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Oops this is a bug in test case not the actual code, good catch. This
> problem is not unique to Java 7. The problem only appears in Java 7 because
> the java serialization tests only use 1 input value. Java 6's put method in
> TreeMap does not call compare when adding to an empty map while Java 7's put
> method does call compare when adding to an empty map.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-105) testJavaSerialization methods fail
in Java 7 because they do not set a group comparator
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259344#comment-13259344 ]
Jim Donofrio commented on MRUNIT-105:
-------------------------------------
add more than one input in 3 testJavaSerialization methods in MapReduceDriver test (both api's) and in PipelineMapReduceDriver test to make test fail in Java 6
set a Integer Comparator as the output group comparator
committed in 1329030
> testJavaSerialization methods fail in Java 7 because they do not set a group comparator
> ---------------------------------------------------------------------------------------
>
> Key: MRUNIT-105
> URL: https://issues.apache.org/jira/browse/MRUNIT-105
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.WritableComparable
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
> at java.util.TreeMap.compare(TreeMap.java:1188)
> at java.util.TreeMap.put(TreeMap.java:531)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
> at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
> at
> org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
> public int compare(Object a, Object b) {
> return compare((WritableComparable)a, (WritableComparable)b);
> }
> Java 7:
> Entry<K,V> t = root;
> if (t == null) {
> compare(key, key); // type (and possibly null) check
> root = new Entry<>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Java 6:
> Entry<K,V> t = root;
> if (t == null) {
> // TBD:
> // 5045147: (coll) Adding null to an empty TreeSet should
> // throw NullPointerException
> //
> // compare(key, key); // type check
> root = new Entry<K,V>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Oops this is a bug in test case not the actual code, good catch. This
> problem is not unique to Java 7. The problem only appears in Java 7 because
> the java serialization tests only use 1 input value. Java 6's put method in
> TreeMap does not call compare when adding to an empty map while Java 7's put
> method does call compare when adding to an empty map.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-105) testJavaSerialization methods fail
in Java 7 because they do not set a group comparator
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259414#comment-13259414 ]
Dave Beech commented on MRUNIT-105:
-----------------------------------
Nice - thanks for doing the legwork on that one Jim. Sorry I didn't get a chance to look into it further myself before reporting it.
> testJavaSerialization methods fail in Java 7 because they do not set a group comparator
> ---------------------------------------------------------------------------------------
>
> Key: MRUNIT-105
> URL: https://issues.apache.org/jira/browse/MRUNIT-105
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.WritableComparable
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
> at java.util.TreeMap.compare(TreeMap.java:1188)
> at java.util.TreeMap.put(TreeMap.java:531)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
> at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
> at
> org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
> public int compare(Object a, Object b) {
> return compare((WritableComparable)a, (WritableComparable)b);
> }
> Java 7:
> Entry<K,V> t = root;
> if (t == null) {
> compare(key, key); // type (and possibly null) check
> root = new Entry<>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Java 6:
> Entry<K,V> t = root;
> if (t == null) {
> // TBD:
> // 5045147: (coll) Adding null to an empty TreeSet should
> // throw NullPointerException
> //
> // compare(key, key); // type check
> root = new Entry<K,V>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Oops this is a bug in test case not the actual code, good catch. This
> problem is not unique to Java 7. The problem only appears in Java 7 because
> the java serialization tests only use 1 input value. Java 6's put method in
> TreeMap does not call compare when adding to an empty map while Java 7's put
> method does call compare when adding to an empty map.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-105) testJavaSerialization methods fail in
Java 7 because they do not set a group comparator
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-105:
--------------------------------
Summary: testJavaSerialization methods fail in Java 7 because they do not set a group comparator (was: java serialization tests fail because they do not set a group comparator)
> testJavaSerialization methods fail in Java 7 because they do not set a group comparator
> ---------------------------------------------------------------------------------------
>
> Key: MRUNIT-105
> URL: https://issues.apache.org/jira/browse/MRUNIT-105
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.9.0
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 1.0.0
>
>
> From http://mail-archives.apache.org/mod_mbox/incubator-mrunit-dev/201204.mbox/%3C4F927519.2040207%40gmx.net%3E
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> org.apache.hadoop.io.WritableComparable
> at
> org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:119)
> at java.util.TreeMap.compare(TreeMap.java:1188)
> at java.util.TreeMap.put(TreeMap.java:531)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.shuffle(MapReduceDriverBase.java:172)
> at org.apache.hadoop.mrunit.MapReduceDriver.run(MapReduceDriver.java:328)
> at
> org.apache.hadoop.mrunit.MapReduceDriverBase.runTest(MapReduceDriverBase.java:137)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:158)
> at
> org.apache.hadoop.mrunit.TestMapReduceDriver.testJavaSerialization(TestMapReduceDriver.java:409)
> public int compare(Object a, Object b) {
> return compare((WritableComparable)a, (WritableComparable)b);
> }
> Java 7:
> Entry<K,V> t = root;
> if (t == null) {
> compare(key, key); // type (and possibly null) check
> root = new Entry<>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Java 6:
> Entry<K,V> t = root;
> if (t == null) {
> // TBD:
> // 5045147: (coll) Adding null to an empty TreeSet should
> // throw NullPointerException
> //
> // compare(key, key); // type check
> root = new Entry<K,V>(key, value, null);
> size = 1;
> modCount++;
> return null;
> }
> Oops this is a bug in test case not the actual code, good catch. This
> problem is not unique to Java 7. The problem only appears in Java 7 because
> the java serialization tests only use 1 input value. Java 6's put method in
> TreeMap does not call compare when adding to an empty map while Java 7's put
> method does call compare when adding to an empty map.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira