You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Spyros Blanas (JIRA)" <ji...@apache.org> on 2008/06/10 04:01:45 UTC
[jira] Created: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
------------------------------------------------------------------------------------
Key: HADOOP-3522
URL: https://issues.apache.org/jira/browse/HADOOP-3522
Project: Hadoop Core
Issue Type: Bug
Affects Versions: 0.17.0
Reporter: Spyros Blanas
h2.Problem
The ValuesIterator.next() doesn't return a new object representing the next element.
It is expected that this is the case. Example from java.lang.String source code:
{noformat}
public boolean equals(Object anObject) {
if (this == anObject) {
return true;
}
if (anObject instanceof String) {
// custom code here
}
}
{noformat}
Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
h2. What is affected
The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
Manifestations of this bug in 0.17.0:
* The Text class checks for equality similarly to String.equals(), which is shown above.
* The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
h2. JUnit test
Patch against hadoop-0.17.0:
{noformat}
Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
===================================================================
--- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
+++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
@@ -122,4 +122,52 @@
runValueIterator(tmpDir, testCase, conf);
}
}
+
+ public void runValueIterator2(Path tmpDir, Pair[] vals,
+ Configuration conf) throws IOException {
+ FileSystem fs = tmpDir.getFileSystem(conf);
+ Path path = new Path(tmpDir, "data.in");
+ SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
+ Text.class,
+ Text.class);
+ for(Pair p: vals) {
+ writer.append(new Text(p.key), new Text(p.value));
+ }
+ writer.close();
+ SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
+ Text.class, conf);
+ SequenceFile.Sorter.RawKeyValueIterator rawItr =
+ sorter.merge(new Path[]{path}, false, tmpDir);
+ @SuppressWarnings("unchecked") // WritableComparators are not generic
+ ReduceTask.ValuesIterator<Text,Text> valItr =
+ new ReduceTask.ValuesIterator<Text,Text>(rawItr,
+ WritableComparator.get(Text.class), Text.class, Text.class,
+ conf, new NullProgress());
+ while (valItr.more()) {
+ Object key = valItr.getKey();
+ String keyString = key.toString();
+ // make sure it matches!
+ assertEquals(vals[0].key, keyString);
+ // must have at least 1 value!
+ assertTrue(valItr.hasNext());
+ Text value1 = valItr.next();
+ // must have at least 2 values!
+ assertTrue(valItr.hasNext());
+ Text value2 = valItr.next();
+ // do test
+ assertNotSame(value1, value2);
+ assertTrue(!value1.equals(value2));
+ assertTrue(!value2.equals(value1));
+ // make sure the key hasn't changed under the hood
+ assertEquals(keyString, valItr.getKey().toString());
+ valItr.nextKey();
+ }
+ }
+
+ public void testValueIterator2() throws Exception {
+ Path tmpDir = new Path("build/test/test.reduce.task");
+ Configuration conf = new Configuration();
+ Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
+ runValueIterator2(tmpDir, test, conf);
+ }
}
{noformat}
h2. A program with a bug
I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
{noformat}
package test;
import java.io.*;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.util.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.*;
public class TestNewReducer extends Configured implements Tool {
public int run(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("TestNewReducer <inDir> <outDir>");
return -1;
}
JobConf conf = new JobConf(getConf(), TestNewReducer.class);
conf.setJobName("test-newreducer");
conf.setInputFormat(TextInputFormat.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(LongWritable.class);
conf.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
System.exit(res);
}
public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void map (LongWritable key, Text value,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
output.collect(nill, value);
}
}
public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void reduce(LongWritable key, Iterator<Text> values,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
Text t1 = null;
if (values.hasNext())
t1 = values.next(); // t1 is the first element in values
while(values.hasNext()) {
// is this element equal to t1?
Text t = values.next();
output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
}
}
}
}
{noformat}
h2. Possible fixes
# Return a new object each time for next(). This might have significant overhead.
# Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
# Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3522) ValuesIterator.next() doesn't
return a new object, thus failing many equals() tests.
Posted by "Spyros Blanas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603959#action_12603959 ]
Spyros Blanas commented on HADOOP-3522:
---------------------------------------
Much better! I hope that users will notice.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Attachments: reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-3522:
--------------------------------------
Release Note: (was: My mistake. The javadoc modification is not a blocker. The blocker is HADOOP-3526.)
My mistake. The javadoc modification is not a blocker. The blocker is HADOOP-3526.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-3522:
--------------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.17.1
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Priority: Blocker
> Fix For: 0.17.1
>
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3522) ValuesIterator.next() doesn't
return a new object, thus failing many equals() tests.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604103#action_12604103 ]
Hadoop QA commented on HADOOP-3522:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383779/reduce-doc-19.patch
against trunk revision 666340.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2635/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2635/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2635/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2635/console
This message is automatically generated.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3522) ValuesIterator.next()
doesn't return a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603930#action_12603930 ]
owen.omalley edited comment on HADOOP-3522 at 6/10/08 9:59 AM:
----------------------------------------------------------------
This is working as designed.
{quote}
Possible fixes
1. Return a new object each time for next(). This might have significant overhead.
{quote}
This is the problem that was fixed. Forcing the inner loops of map/reduce to allocate objects is a huge cost.
{quote}
2. Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
{quote}
No, this will make the semantics much more confusing.
{quote}
3. Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
{quote}
I don't understand your point. *Any* equals function must return true if this == anObject.
It *should* be documented better in the reduce interface that the iterator returns the same (modified) value each time.
was (Author: owen.omalley):
This is working as designed.
1. Return a new object each time for next(). This might have significant overhead.
2. Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
3. Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
Description
Problem
The ValuesIterator.next() doesn't return a new object representing the next element. It is expected that this is the case. Example from java.lang.String source code:
public boolean equals(Object anObject) {
if (this == anObject) {
return true;
}
if (anObject instanceof String) {
// custom code here
}
}
Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
What is affected
The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected. Manifestations of this bug in 0.17.0:
* The Text class checks for equality similarly to String.equals(), which is shown above.
* The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
JUnit test
Patch against hadoop-0.17.0:
Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
===================================================================
--- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
+++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
@@ -122,4 +122,52 @@
runValueIterator(tmpDir, testCase, conf);
}
}
+
+ public void runValueIterator2(Path tmpDir, Pair[] vals,
+ Configuration conf) throws IOException {
+ FileSystem fs = tmpDir.getFileSystem(conf);
+ Path path = new Path(tmpDir, "data.in");
+ SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
+ Text.class,
+ Text.class);
+ for(Pair p: vals) {
+ writer.append(new Text(p.key), new Text(p.value));
+ }
+ writer.close();
+ SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
+ Text.class, conf);
+ SequenceFile.Sorter.RawKeyValueIterator rawItr =
+ sorter.merge(new Path[]{path}, false, tmpDir);
+ @SuppressWarnings("unchecked") // WritableComparators are not generic
+ ReduceTask.ValuesIterator<Text,Text> valItr =
+ new ReduceTask.ValuesIterator<Text,Text>(rawItr,
+ WritableComparator.get(Text.class), Text.class, Text.class,
+ conf, new NullProgress());
+ while (valItr.more()) {
+ Object key = valItr.getKey();
+ String keyString = key.toString();
+ // make sure it matches!
+ assertEquals(vals[0].key, keyString);
+ // must have at least 1 value!
+ assertTrue(valItr.hasNext());
+ Text value1 = valItr.next();
+ // must have at least 2 values!
+ assertTrue(valItr.hasNext());
+ Text value2 = valItr.next();
+ // do test
+ assertNotSame(value1, value2);
+ assertTrue(!value1.equals(value2));
+ assertTrue(!value2.equals(value1));
+ // make sure the key hasn't changed under the hood
+ assertEquals(keyString, valItr.getKey().toString());
+ valItr.nextKey();
+ }
+ }
+
+ public void testValueIterator2() throws Exception {
+ Path tmpDir = new Path("build/test/test.reduce.task");
+ Configuration conf = new Configuration();
+ Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
+ runValueIterator2(tmpDir, test, conf);
+ }
}
A program with a bug
I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
package test;
import java.io.*;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.util.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.*;
public class TestNewReducer extends Configured implements Tool {
public int run(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("TestNewReducer <inDir> <outDir>");
return -1;
}
JobConf conf = new JobConf(getConf(), TestNewReducer.class);
conf.setJobName("test-newreducer");
conf.setInputFormat(TextInputFormat.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(LongWritable.class);
conf.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
System.exit(res);
}
public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void map (LongWritable key, Text value,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
output.collect(nill, value);
}
}
public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void reduce(LongWritable key, Iterator<Text> values,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
Text t1 = null;
if (values.hasNext())
t1 = values.next(); // t1 is the first element in values
while(values.hasNext()) {
// is this element equal to t1?
Text t = values.next();
output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
}
}
}
}
{quote}
Possible fixes
1. Return a new object each time for next(). This might have significant overhead.
{quote}
This is the problem that was fixed. Forcing the inner loops of map/reduce to allocate objects is a huge cost.
{quote}
2. Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
{quote}
No, this will make the semantics much more confusing.
{quote}
3. Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
{quote}
I don't understand your point. *Any* equals function must return true if this == anObject.
It *should* be documented better in the reduce interface that the iterator returns the same (modified) value each time.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3522:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.17.1
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Fix For: 0.17.1
>
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3522) ValuesIterator.next() doesn't
return a new object, thus failing many equals() tests.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605943#action_12605943 ]
Hudson commented on HADOOP-3522:
--------------------------------
Integrated in Hadoop-trunk #522 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/522/])
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Fix For: 0.17.1
>
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-3522.
-----------------------------------
Resolution: Won't Fix
This is working as designed.
1. Return a new object each time for next(). This might have significant overhead.
2. Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
3. Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
Description
Problem
The ValuesIterator.next() doesn't return a new object representing the next element. It is expected that this is the case. Example from java.lang.String source code:
public boolean equals(Object anObject) {
if (this == anObject) {
return true;
}
if (anObject instanceof String) {
// custom code here
}
}
Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
What is affected
The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected. Manifestations of this bug in 0.17.0:
* The Text class checks for equality similarly to String.equals(), which is shown above.
* The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
JUnit test
Patch against hadoop-0.17.0:
Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
===================================================================
--- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
+++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
@@ -122,4 +122,52 @@
runValueIterator(tmpDir, testCase, conf);
}
}
+
+ public void runValueIterator2(Path tmpDir, Pair[] vals,
+ Configuration conf) throws IOException {
+ FileSystem fs = tmpDir.getFileSystem(conf);
+ Path path = new Path(tmpDir, "data.in");
+ SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
+ Text.class,
+ Text.class);
+ for(Pair p: vals) {
+ writer.append(new Text(p.key), new Text(p.value));
+ }
+ writer.close();
+ SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
+ Text.class, conf);
+ SequenceFile.Sorter.RawKeyValueIterator rawItr =
+ sorter.merge(new Path[]{path}, false, tmpDir);
+ @SuppressWarnings("unchecked") // WritableComparators are not generic
+ ReduceTask.ValuesIterator<Text,Text> valItr =
+ new ReduceTask.ValuesIterator<Text,Text>(rawItr,
+ WritableComparator.get(Text.class), Text.class, Text.class,
+ conf, new NullProgress());
+ while (valItr.more()) {
+ Object key = valItr.getKey();
+ String keyString = key.toString();
+ // make sure it matches!
+ assertEquals(vals[0].key, keyString);
+ // must have at least 1 value!
+ assertTrue(valItr.hasNext());
+ Text value1 = valItr.next();
+ // must have at least 2 values!
+ assertTrue(valItr.hasNext());
+ Text value2 = valItr.next();
+ // do test
+ assertNotSame(value1, value2);
+ assertTrue(!value1.equals(value2));
+ assertTrue(!value2.equals(value1));
+ // make sure the key hasn't changed under the hood
+ assertEquals(keyString, valItr.getKey().toString());
+ valItr.nextKey();
+ }
+ }
+
+ public void testValueIterator2() throws Exception {
+ Path tmpDir = new Path("build/test/test.reduce.task");
+ Configuration conf = new Configuration();
+ Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
+ runValueIterator2(tmpDir, test, conf);
+ }
}
A program with a bug
I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
package test;
import java.io.*;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.util.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.mapred.lib.*;
public class TestNewReducer extends Configured implements Tool {
public int run(String[] args) throws Exception {
if (args.length < 2) {
System.out.println("TestNewReducer <inDir> <outDir>");
return -1;
}
JobConf conf = new JobConf(getConf(), TestNewReducer.class);
conf.setJobName("test-newreducer");
conf.setInputFormat(TextInputFormat.class);
conf.setMapperClass(Map.class);
conf.setReducerClass(Reduce.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setOutputKeyClass(LongWritable.class);
conf.setOutputValueClass(Text.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
return 0;
}
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
System.exit(res);
}
public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void map (LongWritable key, Text value,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
output.collect(nill, value);
}
}
public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
private final static LongWritable nill = new LongWritable(1);
public void reduce(LongWritable key, Iterator<Text> values,
OutputCollector<LongWritable, Text> output,
Reporter reporter) throws IOException {
Text t1 = null;
if (values.hasNext())
t1 = values.next(); // t1 is the first element in values
while(values.hasNext()) {
// is this element equal to t1?
Text t = values.next();
output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
}
}
}
}
{quote}
Possible fixes
1. Return a new object each time for next(). This might have significant overhead.
{quote}
This is the problem that was fixed. Forcing the inner loops of map/reduce to allocate objects is a huge cost.
{quote}
2. Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
{quote}
No, this will make the semantics much more confusing.
{quote}
3. Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
{quote}
I don't understand your point. *Any* equals function must return true if this == anObject.
It *should* be documented better in the reduce interface that the iterator returns the same (modified) value each time.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3522:
----------------------------------
Attachment: reduce-doc-19.patch
The version for 0.18 and 0.19.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reopened HADOOP-3522:
-----------------------------------
Assignee: Owen O'Malley
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar updated HADOOP-3522:
--------------------------------------
Priority: Major (was: Blocker)
Fix Version/s: (was: 0.17.1)
Release Note: My mistake. The javadoc modification is not a blocker. The blocker is HADOOP-3526.
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3522:
----------------------------------
Attachment: reduce-doc.patch
Here is a javadoc patch for the reduce. Does this help?
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Attachments: reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3522) ValuesIterator.next() doesn't return
a new object, thus failing many equals() tests.
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-3522:
----------------------------------
Status: Patch Available (was: Reopened)
> ValuesIterator.next() doesn't return a new object, thus failing many equals() tests.
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-3522
> URL: https://issues.apache.org/jira/browse/HADOOP-3522
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0
> Reporter: Spyros Blanas
> Assignee: Owen O'Malley
> Attachments: reduce-doc-19.patch, reduce-doc.patch
>
>
> h2.Problem
> The ValuesIterator.next() doesn't return a new object representing the next element.
> It is expected that this is the case. Example from java.lang.String source code:
> {noformat}
> public boolean equals(Object anObject) {
> if (this == anObject) {
> return true;
> }
> if (anObject instanceof String) {
> // custom code here
> }
> }
> {noformat}
> Changing the private fields of the object and returning it from next() will make this object fail the equals() test, because this==anObject will always be true.
> h2. What is affected
> The reduce() method presents (a subclass of) ValuesIterator to the user. So every user-defined class can be affected.
> Manifestations of this bug in 0.17.0:
> * The Text class checks for equality similarly to String.equals(), which is shown above.
> * The contrib/data_join breaks because it stores tags in a Map. The behavior of the next() method makes Object.equals() be true for all tags.
> h2. JUnit test
> Patch against hadoop-0.17.0:
> {noformat}
> Index: src/test/org/apache/hadoop/mapred/TestReduceTask.java
> ===================================================================
> --- src/test/org/apache/hadoop/mapred/TestReduceTask.java (revision 665935)
> +++ src/test/org/apache/hadoop/mapred/TestReduceTask.java (working copy)
> @@ -122,4 +122,52 @@
> runValueIterator(tmpDir, testCase, conf);
> }
> }
> +
> + public void runValueIterator2(Path tmpDir, Pair[] vals,
> + Configuration conf) throws IOException {
> + FileSystem fs = tmpDir.getFileSystem(conf);
> + Path path = new Path(tmpDir, "data.in");
> + SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path,
> + Text.class,
> + Text.class);
> + for(Pair p: vals) {
> + writer.append(new Text(p.key), new Text(p.value));
> + }
> + writer.close();
> + SequenceFile.Sorter sorter = new SequenceFile.Sorter(fs, Text.class,
> + Text.class, conf);
> + SequenceFile.Sorter.RawKeyValueIterator rawItr =
> + sorter.merge(new Path[]{path}, false, tmpDir);
> + @SuppressWarnings("unchecked") // WritableComparators are not generic
> + ReduceTask.ValuesIterator<Text,Text> valItr =
> + new ReduceTask.ValuesIterator<Text,Text>(rawItr,
> + WritableComparator.get(Text.class), Text.class, Text.class,
> + conf, new NullProgress());
> + while (valItr.more()) {
> + Object key = valItr.getKey();
> + String keyString = key.toString();
> + // make sure it matches!
> + assertEquals(vals[0].key, keyString);
> + // must have at least 1 value!
> + assertTrue(valItr.hasNext());
> + Text value1 = valItr.next();
> + // must have at least 2 values!
> + assertTrue(valItr.hasNext());
> + Text value2 = valItr.next();
> + // do test
> + assertNotSame(value1, value2);
> + assertTrue(!value1.equals(value2));
> + assertTrue(!value2.equals(value1));
> + // make sure the key hasn't changed under the hood
> + assertEquals(keyString, valItr.getKey().toString());
> + valItr.nextKey();
> + }
> + }
> +
> + public void testValueIterator2() throws Exception {
> + Path tmpDir = new Path("build/test/test.reduce.task");
> + Configuration conf = new Configuration();
> + Pair[] test = new Pair[]{ new Pair("k1", "v1"), new Pair("k1", "v2") };
> + runValueIterator2(tmpDir, test, conf);
> + }
> }
> {noformat}
> h2. A program with a bug
> I would imagine that a programmer would be really confused when everything is equal in the example below, for any text input:
> {noformat}
> package test;
> import java.io.*;
> import java.util.*;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.conf.*;
> import org.apache.hadoop.io.*;
> import org.apache.hadoop.util.*;
> import org.apache.hadoop.mapred.*;
> import org.apache.hadoop.mapred.lib.*;
> public class TestNewReducer extends Configured implements Tool {
> public int run(String[] args) throws Exception {
> if (args.length < 2) {
> System.out.println("TestNewReducer <inDir> <outDir>");
> return -1;
> }
> JobConf conf = new JobConf(getConf(), TestNewReducer.class);
> conf.setJobName("test-newreducer");
> conf.setInputFormat(TextInputFormat.class);
> conf.setMapperClass(Map.class);
> conf.setReducerClass(Reduce.class);
> conf.setOutputFormat(TextOutputFormat.class);
> conf.setOutputKeyClass(LongWritable.class);
> conf.setOutputValueClass(Text.class);
> FileInputFormat.setInputPaths(conf, new Path(args[0]));
> FileOutputFormat.setOutputPath(conf, new Path(args[1]));
> JobClient.runJob(conf);
> return 0;
> }
> public static void main(String[] args) throws Exception {
> int res = ToolRunner.run(new Configuration(), new TestNewReducer(), args);
> System.exit(res);
> }
> public static class Map extends MapReduceBase implements Mapper <LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void map (LongWritable key, Text value,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> output.collect(nill, value);
> }
> }
> public static class Reduce extends MapReduceBase implements Reducer<LongWritable, Text, LongWritable, Text> {
> private final static LongWritable nill = new LongWritable(1);
> public void reduce(LongWritable key, Iterator<Text> values,
> OutputCollector<LongWritable, Text> output,
> Reporter reporter) throws IOException {
> Text t1 = null;
> if (values.hasNext())
> t1 = values.next(); // t1 is the first element in values
> while(values.hasNext()) {
> // is this element equal to t1?
> Text t = values.next();
> output.collect(nill, t.equals(t1) ? new Text("Equal") : new Text("Not equal"));
> }
> }
> }
> }
> {noformat}
> h2. Possible fixes
> # Return a new object each time for next(). This might have significant overhead.
> # Return a new object for unknown object types and reuse the same object for known types (like Text). Remove "if (this==anObject) return true;" check from all equals() methods for known objects.
> # Document clearly that all user-defined classes must implement an equals() method, which doesn't do the "if (this==anObject) return true;" check (ie. push the problem to the user).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.