You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Daniel Templeton (JIRA)" <ji...@apache.org> on 2017/01/26 15:10:24 UTC
[jira] [Created] (MAPREDUCE-6837) Add an equivalent to Crunch's
Pair class
Daniel Templeton created MAPREDUCE-6837:
-------------------------------------------
Summary: Add an equivalent to Crunch's Pair class
Key: MAPREDUCE-6837
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6837
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: Daniel Templeton
Crunch has this great {{Pair}} class (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that save you from constantly implementing composite writables. It seems silly that we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable<P extends WritableComparable, S extends WritableComparable> implements WritableComparable<CompositeWritable> {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a custom grouping comparator. That comparator could be implemented as part of this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I don't think we need to add equivalents. If someone really wants that capability, they can nest composite keys.
Don't forget to add unit tests!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org