You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Justin Patterson (JIRA)" <ji...@apache.org> on 2009/07/07 21:36:14 UTC
[jira] Created: (HADOOP-6129) MapFile doesn't worh with
serializables other than Writables
MapFile doesn't worh with serializables other than Writables
------------------------------------------------------------
Key: HADOOP-6129
URL: https://issues.apache.org/jira/browse/HADOOP-6129
Project: Hadoop Common
Issue Type: Improvement
Components: io, mapred
Affects Versions: 0.20.0
Reporter: Justin Patterson
Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6162) MapFile doesn't work with
serializables other than Writables
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732972#action_12732972 ]
Chris Douglas commented on HADOOP-6162:
---------------------------------------
On second thought, this issue should be in common; the MapFileOutputFormat changes can be moved to a MR issue.
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6162
> URL: https://issues.apache.org/jira/browse/HADOOP-6162
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Justin Patterson
> Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6162) MapFile doesn't work with
serializables other than Writables
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735517#action_12735517 ]
Sharad Agarwal commented on HADOOP-6162:
----------------------------------------
Few comments on the patch:
There are lot of javac warnings. We should get rid of them.
Tests for ArrayFile, BloomMapFile and SetFile also should be added to their respective junit tests.
Should we deprecate constructors takings Writables ?
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6162
> URL: https://issues.apache.org/jira/browse/HADOOP-6162
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Justin Patterson
> Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6129) MapFile doesn't work with
serializables other than Writables
Posted by "Justin Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Patterson updated HADOOP-6129:
-------------------------------------
Status: Patch Available (was: Open)
Here's the patch that I used to get this working in my own app.
It basically uses the SerializationFactory for key/value serialization. The key comparator is either specified by the Writer creator or, for legacy reasons, is created through WritableComparator.get().
The MapFileOutputFormat creates the key comparator using job.getMapOutputKeyComparator().
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6129
> URL: https://issues.apache.org/jira/browse/HADOOP-6129
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, mapred
> Affects Versions: 0.20.0
> Reporter: Justin Patterson
> Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6129) MapFile doesn't work with
serializables other than Writables
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728374#action_12728374 ]
Hadoop QA commented on HADOOP-6129:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12412780/HADOOP-6129.patch
against trunk revision 791937.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/556/console
This message is automatically generated.
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6129
> URL: https://issues.apache.org/jira/browse/HADOOP-6129
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, mapred
> Affects Versions: 0.20.0
> Reporter: Justin Patterson
> Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6129) MapFile doesn't work with
serializables other than Writables
Posted by "Justin Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Patterson updated HADOOP-6129:
-------------------------------------
Attachment: HADOOP-6129.patch
Patch attached.
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6129
> URL: https://issues.apache.org/jira/browse/HADOOP-6129
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, mapred
> Affects Versions: 0.20.0
> Reporter: Justin Patterson
> Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6162) MapFile doesn't work with
serializables other than Writables
Posted by "Justin Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Patterson updated HADOOP-6162:
-------------------------------------
Attachment: (was: HADOOP-6129.patch)
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6162
> URL: https://issues.apache.org/jira/browse/HADOOP-6162
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Justin Patterson
> Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6129) MapFile doesn't work with
serializables other than Writables
Posted by "Justin Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Patterson updated HADOOP-6129:
-------------------------------------
Summary: MapFile doesn't work with serializables other than Writables (was: MapFile doesn't worh with serializables other than Writables)
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6129
> URL: https://issues.apache.org/jira/browse/HADOOP-6129
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, mapred
> Affects Versions: 0.20.0
> Reporter: Justin Patterson
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6162) MapFile doesn't work with
serializables other than Writables
Posted by "Justin Patterson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Justin Patterson updated HADOOP-6162:
-------------------------------------
Attachment: HADOOP-6162.patch
Here's another patch that works with the new (divided) file structure and includes a unit test to prove that it works.
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6162
> URL: https://issues.apache.org/jira/browse/HADOOP-6162
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Justin Patterson
> Attachments: HADOOP-6162.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Moved: (HADOOP-6162) MapFile doesn't work with serializables
other than Writables
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas moved MAPREDUCE-738 to HADOOP-6162:
-------------------------------------------------
Key: HADOOP-6162 (was: MAPREDUCE-738)
Project: Hadoop Common (was: Hadoop Map/Reduce)
> MapFile doesn't work with serializables other than Writables
> ------------------------------------------------------------
>
> Key: HADOOP-6162
> URL: https://issues.apache.org/jira/browse/HADOOP-6162
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Justin Patterson
> Attachments: HADOOP-6129.patch
>
>
> Since 0.18 (I think), SequenceFiles have supported serializing arbitrary objects through the serialization framework. MapFiles still don't. They require WritableComparable keys and Writable values.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.