You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tahir Hashmi (JIRA)" <ji...@apache.org> on 2007/04/02 20:52:32 UTC
[jira] Updated: (HADOOP-1001) the output of the map is not type
checked against the specified types
[ https://issues.apache.org/jira/browse/HADOOP-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tahir Hashmi updated HADOOP-1001:
---------------------------------
Attachment: TestMapOutputType.java
The problem with this test is that the exception thrown by MapTask doesn't get propagated to the client. The best effort here is to just make sure that the test cases fail for whatever reason when wrong key or value types are specified and pass with correct types. Hopefully the earlier runs would've failed due to type mismatch exceptions rather than something else. Attaching the revised version of the test now. Here's the kind of output it produces in the logs:
$ cat build/test/TEST-org.apache.hadoop.mapred.TestMapOutputType.txt
Testsuite: org.apache.hadoop.mapred.TestMapOutputType
Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 5.585 sec
------------- Standard Output ---------------
2007-04-03 00:18:02,162 WARN util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(51)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2007-04-03 00:18:02,508 INFO mapred.InputFormatBase (InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:03,238 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(166)) - job_bzpgha
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:312)
at org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:04,121 INFO mapred.InputFormatBase (InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:04,249 WARN mapred.LocalJobRunner (LocalJobRunner.java:run(166)) - job_4f7xh3
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:317)
at org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:05,324 INFO mapred.InputFormatBase (InputFormatBase.java:validateInput(141)) - Total input paths to process : 1
2007-04-03 00:18:05,604 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - file:/home/tahir/Desktop/Hadoop/build/test/test.mapred.spill/in/part0:0+150
2007-04-03 00:18:05,752 INFO mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) - reduce > reduce
------------- ---------------- ---------------
Testcase: testKeyMismatch took 3.144 sec
Testcase: testValueMismatch took 1.128 sec
Testcase: testNoMismatch took 1.302 sec
> the output of the map is not type checked against the specified types
> ---------------------------------------------------------------------
>
> Key: HADOOP-1001
> URL: https://issues.apache.org/jira/browse/HADOOP-1001
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.0
> Reporter: Owen O'Malley
> Assigned To: Tahir Hashmi
> Fix For: 0.13.0
>
> Attachments: 1001.patch, 1001.patch, 1001.patch, TestMapOutputType.java, TestMapOutputType.java
>
>
> The output of the map is not checked against the types specified in the JobConf leading to hard to diagnose bugs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.