You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2008/09/03 06:31:44 UTC
[jira] Created: (HADOOP-4057) MultipleFileInputFormat should
support a sub InputFormat
MultipleFileInputFormat should support a sub InputFormat
--------------------------------------------------------
Key: HADOOP-4057
URL: https://issues.apache.org/jira/browse/HADOOP-4057
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Owen O'Malley
Currently, to use MultipleFileInputFormat you need to create a subclass, but it seems pretty easy to modify it to use a sub InputFormat and put the results together into bigger splits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4057) MultipleFileInputFormat should
support a sub InputFormat
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12627899#action_12627899 ]
Owen O'Malley commented on HADOOP-4057:
---------------------------------------
I'd suggest something like this:
{code}
public class MultipleFileInputFormat implements InputFormat {
public static void setUserInputFormat(Configuration conf, Class<? extends InputFormat> inputFormat);
public static Class<? extends InputFormat> getUserInputFormat(Configuration conf);
public InputSplit[] getSplits(JobConf conf, int maps) {
InputFormat userInputFormat = ReflectionUtils.newInstance(getUserInputFormat(conf), conf);
InputSplit[] userSplits = userInputFormat.getSPlits(conf, maps);
... merge them together ...
return exactly maps splits
}
public RecordReader getRecordReader(InputSplit split, JobConf conf, Reporter reporter) {
.. return a record reader that returns all of the records for split 1, split 2, etc..
}
}
{code}
> MultipleFileInputFormat should support a sub InputFormat
> --------------------------------------------------------
>
> Key: HADOOP-4057
> URL: https://issues.apache.org/jira/browse/HADOOP-4057
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
>
> Currently, to use MultipleFileInputFormat you need to create a subclass, but it seems pretty easy to modify it to use a sub InputFormat and put the results together into bigger splits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.