You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2008/03/11 12:46:49 UTC
[jira] Created: (HADOOP-2997) Add test for non-writable serializer
Add test for non-writable serializer
------------------------------------
Key: HADOOP-2997
URL: https://issues.apache.org/jira/browse/HADOOP-2997
Project: Hadoop Core
Issue Type: Test
Components: mapred
Reporter: Tom White
Assignee: Tom White
Fix For: 0.17.0
It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-2997:
------------------------------
Attachment: hadoop-2997-v2.patch
I've fixed the release audit problem, the failing unit test reported by hudson was unrelated to this patch. *However*, this test now hangs consistently - it looks like it's related to the changes in HADOOP-2399 since the test passes when I run against revision 636622.
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-2997:
------------------------------
Attachment: hadoop-2997.patch
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582928#action_12582928 ]
Owen O'Malley commented on HADOOP-2997:
---------------------------------------
+1
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583309#action_12583309 ]
Hudson commented on HADOOP-2997:
--------------------------------
Integrated in Hadoop-trunk #445 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/445/])
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-2997:
------------------------------
Status: Patch Available (was: Open)
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2997:
--------------------------------
Status: Open (was: Patch Available)
Tom, could you please address the hudson warning and resubmit the patch. Thanks!
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578693#action_12578693 ]
Hadoop QA commented on HADOOP-2997:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377777/hadoop-2997-v3.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit +1. The applied patch does not generate any new release audit warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/console
This message is automatically generated.
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12582996#action_12582996 ]
Hudson commented on HADOOP-2997:
--------------------------------
Integrated in Hadoop-trunk #444 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/444/])
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-2997:
------------------------------
Status: Patch Available (was: Open)
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578052#action_12578052 ]
Hadoop QA commented on HADOOP-2997:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377604/hadoop-2997.patch
against trunk revision 619744.
@author +1. The patch does not contain any @author tags.
tests included +1. The patch appears to include 3 new or modified tests.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new javac compiler warnings.
release audit -1. The applied patch generated 193 release audit warnings (more than the trunk's current 192 warnings).
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/build/test/checkstyle-errors.html
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/current/releaseAuditDiffWarnings.txt
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/console
This message is automatically generated.
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-2997:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Tom!
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Priority: Blocker
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2997) Add test for non-writable
serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578250#action_12578250 ]
Tom White commented on HADOOP-2997:
-----------------------------------
Tracked down the problem - the change in HADOOP-2399 creates instances of keys and values to be reused, however this is not possible for Java serialization since the types don't necessarily have no-arg constructors:
java.lang.RuntimeException: java.lang.NoSuchMethodException: java.lang.Long.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.<init>(ReduceTask.java:213)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.<init>(ReduceTask.java:306)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:388)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2073)
Caused by: java.lang.NoSuchMethodException: java.lang.Long.<init>()
at java.lang.Class.getConstructor0(Class.java:2678)
at java.lang.Class.getDeclaredConstructor(Class.java:1953)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
... 4 more
The fix is to get the deserializer to create the key and value types, if needed. I'm testing a patch.
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2997) Add test for non-writable serializer
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-2997:
------------------------------
Attachment: hadoop-2997-v3.patch
> Add test for non-writable serializer
> ------------------------------------
>
> Key: HADOOP-2997
> URL: https://issues.apache.org/jira/browse/HADOOP-2997
> Project: Hadoop Core
> Issue Type: Test
> Components: mapred
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.17.0
>
> Attachments: hadoop-2997-v2.patch, hadoop-2997-v3.patch, hadoop-2997.patch
>
>
> It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.