You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2006/08/11 22:29:14 UTC
[jira] Created: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Remove the need for users to specify the types of the inputs
------------------------------------------------------------
Key: HADOOP-450
URL: http://issues.apache.org/jira/browse/HADOOP-450
Project: Hadoop
Issue Type: Improvement
Components: mapred
Affects Versions: 0.5.0
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
Fix For: 0.6.0
Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
WritableComparable createKey();
Writable createValue();
Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=comments#action_12427986 ]
Doug Cutting commented on HADOOP-450:
-------------------------------------
This issue is not about specifying multiple map functions per job. That should be a separate issue. This is just about moving the ownership of the key and value types from the job to the RecordReader.
I don't understand your objections to the proposed new RecordReader methods createKey() and createValue(). The RecordReader is created by InputFormat.getRecordReader(), which is passed a JobConf, so the RecordReader has access to a JobConf when it creates key and value instances. The RecordReader can then use ReflectionUtils.newInstance() to create configured instances.
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=all ]
Owen O'Malley updated HADOOP-450:
---------------------------------
Status: Patch Available (was: Open)
This patch does:
1. depriciates the JobConf methods for get/set key/value type
2. adds createKey/Value to the RecordReader
3. changes TextInputFormat to use Text rather than UTF8
4. cleans up the uses with examples, lib, and streaming.
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=all ]
Doug Cutting updated HADOOP-450:
--------------------------------
Status: Resolved (was: Patch Available)
Resolution: Fixed
I just committed this, with a warning about the incompatible nature of the changes.
Thanks, Owen!
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
> Attachments: no-input-type.patch
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=comments#action_12427980 ]
Runping Qi commented on HADOOP-450:
-----------------------------------
It seems to me that it is better to add getInputKeyClass/ValueClass methods to RecordReader, since creating an object of these class needs a
reference to JobConf object and RecordReader does not have such a reference.
The MapRunner class can create the key/value objects by calling ReflectionUtil. MapRunner would look like:
public void run(RecordReader input, OutputCollector output,
Reporter reporter)
throws IOException {
try {
// allocate key & value instances that are re-used for all entries
this.inputKeyClass = input.getKeyClass();
this.inputValueClass = input.getValueClass();
WritableComparable key =
(WritableComparable)ReflectionUtils.newInstance(inputKeyClass, job);
Writable value = (Writable)ReflectionUtils.newInstance(inputValueClass,
job);
Class mapperClass = job.getMapperClassFor(this.inputKeyClass, this.inputValueClass);
this.mapper = (Mapper)ReflectionUtils.newInstance(mapperClass, job);
while (input.next(key, value)) {
// map pair to output
mapper.map(key, value, output, reporter);
}
} finally {
mapper.close();
}
}
In the above code, the mapper class is obtained from the job object
through a new method of JobConf class:
Class getMapperClassFor(this.inputKeyClass, this.inputValueClass);
With these, and a few other minor changes, this patch will address jira issue 372 as well.
The other changes include:
add to JobConf class the following methods:
public InputFormat getInputFormat(Path p)
public void setInputFormat(Class theClass, Path p)
public Class getMapperClassFor(Class keyClass, Class valueClass) {
public void setMapperClassFor(Class theClass, Class keyClass, Class valueClass)
replace thefollowing in MapTask class:
final RecordReader rawIn = // open input
job.getInputFormat().getRecordReader
(FileSystem.get(job), split, job, reporter);
with
final RecordReader rawIn = // open input
job.getInputFormat(split.getPath()).getRecordReader
(FileSystem.get(job), split, job, reporter);
With these changes, the application will specify a MapReduce job in the following way:
Configuration defaults = new Configuration();
JobConf theJob = new JobConf(defaults, My.class);
theJob.addInputPath(myInputPath_1)
theJob.setInputFormat(SequenceFileInputFormat, myInputPath_1);
theJob.addInputPath(myInputPath_2)
theJob.setInputFormat(TextInputFormat, myInputPath_2);
theJob.addInputPath(myInputPath_3)
theJob.setInputFormat(SequenceFileInputFormat, myInputPath_3);
theJob.setMapperClassFor(MapperClass_a, LongWritable.class, Text.class);
theJob.setMapperClassFor(MapperClass_b, Key_1.class, Value_1.class);
theJob.setMapperClassFor(MapperClass_c, Key_2.class, Value_2.class);
....
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=all ]
Doug Cutting updated HADOOP-450:
--------------------------------
Status: Open (was: Patch Available)
Um, there's actually no patch attached to this issue.
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=comments#action_12427998 ]
Runping Qi commented on HADOOP-450:
-----------------------------------
I don't really object to add creatKey/Value methods to RecordReader. One way or another is fine. To add these methods, the RecordReader class needs to introduce a reference to conf object, which Owen did, and it is just fine.
I agree we should have a separate patch for Issue 372. With the patch for Issue 450, the patch for Issue 372 will be very simple.
If no body is working on Issue 372, I'd be happy to provide a patch for it.
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
> Attachments: no-input-type.patch
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=all ]
Owen O'Malley updated HADOOP-450:
---------------------------------
Status: Patch Available (was: Open)
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
> Attachments: no-input-type.patch
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-450) Remove the need for users to specify
the types of the inputs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-450?page=all ]
Owen O'Malley updated HADOOP-450:
---------------------------------
Attachment: no-input-type.patch
oops, here is the patch
> Remove the need for users to specify the types of the inputs
> ------------------------------------------------------------
>
> Key: HADOOP-450
> URL: http://issues.apache.org/jira/browse/HADOOP-450
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.5.0
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Fix For: 0.6.0
>
> Attachments: no-input-type.patch
>
>
> Currently, the application specifies the types of the input keys and values and the RecordReader checks them for consistency. It would make more sense to have the RecordReader define the types of keys that it will produce. Therefore, I propose that we add two new methods to RecordReader:
> WritableComparable createKey();
> Writable createValue();
> Note that I propose adding them to the RecordReader rather than the InputFormat, so that they can specific to a particular input split.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira