You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Daniel Templeton (JIRA)" <ji...@apache.org> on 2017/01/26 16:46:24 UTC
[jira] [Updated] (MAPREDUCE-6837) Add an equivalent to Crunch's
Pair class
[ https://issues.apache.org/jira/browse/MAPREDUCE-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Templeton updated MAPREDUCE-6837:
----------------------------------------
Description:
Crunch has this great {{Pair}} class (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that saves one from constantly implementing composite writables. It seems silly that we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable<P extends WritableComparable, S extends WritableComparable> implements WritableComparable<CompositeWritable> {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a custom grouping comparator. That comparator could also be implemented as part of this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I don't think we need to add equivalents. If someone really wants that capability, they can nest composite keys.
Don't forget to add unit tests!
was:
Crunch has this great {{Pair}} class (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that save you from constantly implementing composite writables. It seems silly that we still don't have an equivalent in MR.
I would like to see a new class with the following API:
{code}
package org.apache.hadoop.io;
public class CompositeWritable<P extends WritableComparable, S extends WritableComparable> implements WritableComparable<CompositeWritable> {
public CompositeWritable(P primary, S secondary);
public P getPrimary();
public void setPrimary(P primary);
public S getSecondary();
public void setSecondary(S secondary);
// Return true if both primaries and both secondaries are equal
public boolean equals(CompositeWritable o);
// Return the primary's hash code
public long hashCode();
// Sort first by primary and then by secondary
public int compareTo(CompositeWritable o);
public void readFields(DataInput in);
public void write(DataOutput out);
}
{code}
With such a class, implementing a secondary sort would mean just implementing a custom grouping comparator. That comparator could be implemented as part of this JIRA:
{code}
package org.apache.hadoop.io;
public class CompositeGroupingComparator extends WritableComparator {
...
}
{code}
Or some such.
Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I don't think we need to add equivalents. If someone really wants that capability, they can nest composite keys.
Don't forget to add unit tests!
> Add an equivalent to Crunch's Pair class
> ----------------------------------------
>
> Key: MAPREDUCE-6837
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6837
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Daniel Templeton
> Labels: newbie++
>
> Crunch has this great {{Pair}} class (https://crunch.apache.org/apidocs/0.14.0/org/apache/crunch/Pair.html) that saves one from constantly implementing composite writables. It seems silly that we still don't have an equivalent in MR.
> I would like to see a new class with the following API:
> {code}
> package org.apache.hadoop.io;
> public class CompositeWritable<P extends WritableComparable, S extends WritableComparable> implements WritableComparable<CompositeWritable> {
> public CompositeWritable(P primary, S secondary);
> public P getPrimary();
> public void setPrimary(P primary);
> public S getSecondary();
> public void setSecondary(S secondary);
> // Return true if both primaries and both secondaries are equal
> public boolean equals(CompositeWritable o);
> // Return the primary's hash code
> public long hashCode();
> // Sort first by primary and then by secondary
> public int compareTo(CompositeWritable o);
> public void readFields(DataInput in);
> public void write(DataOutput out);
> }
> {code}
> With such a class, implementing a secondary sort would mean just implementing a custom grouping comparator. That comparator could also be implemented as part of this JIRA:
> {code}
> package org.apache.hadoop.io;
> public class CompositeGroupingComparator extends WritableComparator {
> ...
> }
> {code}
> Or some such.
> Crunch also provides {{Tuple3}}, {{Tuple4}}, and {{TupleN}} classes, but I don't think we need to add equivalents. If someone really wants that capability, they can nest composite keys.
> Don't forget to add unit tests!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org