You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Harsh J (JIRA)" <ji...@apache.org> on 2012/08/22 15:45:43 UTC
[jira] [Created] (MAPREDUCE-4574) Fix TotalOrderParitioner to work
with non-WritableComparable key types
Harsh J created MAPREDUCE-4574:
----------------------------------
Summary: Fix TotalOrderParitioner to work with non-WritableComparable key types
Key: MAPREDUCE-4574
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
That is, since we do not use the values at all (NullWritable), we may as well do:
{code}
private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
Configuration conf) throws IOException {
…
while ((key = (K) reader.next(key)) != null) {
parts.add(key);
key = ReflectionUtils.newInstance(keyClass, conf);
}
…
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472400#comment-13472400 ]
Hudson commented on MAPREDUCE-4574:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1221 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/])
MAPREDUCE-4574. Fix TotalOrderParitioner to work with non-WritableComparable key types. Contributed by Harsh J. (harsh) (Revision 1395936)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395936
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451388#comment-13451388 ]
Harsh J commented on MAPREDUCE-4574:
------------------------------------
Reviews please?
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4574) Fix TotalOrderParitioner to work
with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J reassigned MAPREDUCE-4574:
----------------------------------
Assignee: (was: Harsh J)
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4574) Fix TotalOrderParitioner to work
with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4574:
-------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Doug! I went ahead and committed this to trunk.
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4574) Fix TotalOrderParitioner to work
with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J reassigned MAPREDUCE-4574:
----------------------------------
Assignee: Harsh J
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472256#comment-13472256 ]
Hudson commented on MAPREDUCE-4574:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2834 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2834/])
MAPREDUCE-4574. Fix TotalOrderParitioner to work with non-WritableComparable key types. Contributed by Harsh J. (harsh) (Revision 1395936)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395936
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472343#comment-13472343 ]
Hudson commented on MAPREDUCE-4574:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1190 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/])
MAPREDUCE-4574. Fix TotalOrderParitioner to work with non-WritableComparable key types. Contributed by Harsh J. (harsh) (Revision 1395936)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395936
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472255#comment-13472255 ]
Hudson commented on MAPREDUCE-4574:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2896 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2896/])
MAPREDUCE-4574. Fix TotalOrderParitioner to work with non-WritableComparable key types. Contributed by Harsh J. (harsh) (Revision 1395936)
Result = SUCCESS
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395936
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471773#comment-13471773 ]
Doug Cutting commented on MAPREDUCE-4574:
-----------------------------------------
+1 Change looks good to me.
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472266#comment-13472266 ]
Hudson commented on MAPREDUCE-4574:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2857 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2857/])
MAPREDUCE-4574. Fix TotalOrderParitioner to work with non-WritableComparable key types. Contributed by Harsh J. (harsh) (Revision 1395936)
Result = FAILURE
harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395936
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4574) Fix TotalOrderParitioner to
work with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471282#comment-13471282 ]
Harsh J commented on MAPREDUCE-4574:
------------------------------------
Ping x2? Patch still applies and lets users of Avro, etc. reuse this component without having to re-implement it.
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Assignee: Harsh J
> Priority: Trivial
> Attachments: MAPREDUCE-4574.patch, MAPREDUCE-4574.patch
>
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4574) Fix TotalOrderParitioner to work
with non-WritableComparable key types
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4574:
-------------------------------
Priority: Trivial (was: Major)
> Fix TotalOrderParitioner to work with non-WritableComparable key types
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-4574
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4574
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 2.0.0-alpha
> Reporter: Harsh J
> Priority: Trivial
>
> The current TotalOrderPartitioner class will not work with an alternative serialization library such as Avro.
> To make it work, we may edit the readPartitions bits in it to support non-WritableComparable keys and also remove the WritableComparable check in the class types definition.
> That is, since we do not use the values at all (NullWritable), we may as well do:
> {code}
> private K[] readPartitions(FileSystem fs, Path p, Class<K> keyClass,
> Configuration conf) throws IOException {
> …
> while ((key = (K) reader.next(key)) != null) {
> parts.add(key);
> key = ReflectionUtils.newInstance(keyClass, conf);
> }
> …
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira