You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by maha <ma...@umail.ucsb.edu> on 2010/11/26 23:08:01 UTC
InputSplit is confusing me .. Any clarifications ??
Hi Everyone,
What I really wish for Thanksgiving is some one giving me clarification of how the inputSplit is working.
Eg.
public void map(LongWritable key, Text value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException {
String line = value.toString();
How did we know that map in this case is taking a line and not the whole input document ?
Happy Thanksgiving everyone,
Maha
Re: InputSplit is confusing me .. Any clarifications ??
Posted by Steve Lewis <lo...@gmail.com>.
override *TextInputFormat and have *protected boolean *isSplitable*(
FileSystem<http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/fs/FileSystem.html>
fs,
Path
<http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/fs/Path.html>
filename) return false
This forces read of the entire file
On Sat, Nov 27, 2010 at 9:27 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
> Moving to mapreduce-user@, bcc common-user@. Please use project specific
> lists.
>
> Your InputSplits are defined by your InputFormat. Take a look 'getSplits'
> method in InputFormat.java.
>
> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Input
>
> http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#InputSplit
>
> Arun
>
>
> On Nov 27, 2010, at 3:42 PM, maha wrote:
>
> Sorry I mistyped LineRecordReader by LineInputSplit ... So here is my
>> question again ..
>>
>>
>>> Thanks for the reply .. although I read it has to do with "InputSplit"
>>> which represents the data to be processed by an individual Mapper. By
>>> default it's a LineRecordReader.
>>>
>>> How can I change this property to be FileInputSpilt and Record is the
>>> whole File ?
>>>
>>> something like JobConf.set ("File.input.format","FileInptSplit");
>>>
>>> Is there such way?
>>>
>>> Thanks in advance,
>>> Maha
>>>
>>>
>>
>
--
Steven M. Lewis PhD
4221 105th Ave Ne
Kirkland, WA 98033
206-384-1340 (cell)
Institute for Systems Biology
Seattle WA
Re: InputSplit is confusing me .. Any clarifications ??
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Moving to mapreduce-user@, bcc common-user@. Please use project
specific lists.
Your InputSplits are defined by your InputFormat. Take a look
'getSplits' method in InputFormat.java.
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Input
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#InputSplit
Arun
On Nov 27, 2010, at 3:42 PM, maha wrote:
> Sorry I mistyped LineRecordReader by LineInputSplit ... So here is
> my question again ..
>
>>
>> Thanks for the reply .. although I read it has to do with
>> "InputSplit" which represents the data to be processed by an
>> individual Mapper. By default it's a LineRecordReader.
>>
>> How can I change this property to be FileInputSpilt and Record is
>> the whole File ?
>>
>> something like JobConf.set ("File.input.format","FileInptSplit");
>>
>> Is there such way?
>>
>> Thanks in advance,
>> Maha
>>
>
Re: InputSplit is confusing me .. Any clarifications ??
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Moving to mapreduce-user@, bcc common-user@. Please use project
specific lists.
Your InputSplits are defined by your InputFormat. Take a look
'getSplits' method in InputFormat.java.
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#Job+Input
http://hadoop.apache.org/common/docs/r0.20.0/mapred_tutorial.html#InputSplit
Arun
On Nov 27, 2010, at 3:42 PM, maha wrote:
> Sorry I mistyped LineRecordReader by LineInputSplit ... So here is
> my question again ..
>
>>
>> Thanks for the reply .. although I read it has to do with
>> "InputSplit" which represents the data to be processed by an
>> individual Mapper. By default it's a LineRecordReader.
>>
>> How can I change this property to be FileInputSpilt and Record is
>> the whole File ?
>>
>> something like JobConf.set ("File.input.format","FileInptSplit");
>>
>> Is there such way?
>>
>> Thanks in advance,
>> Maha
>>
>
Re: InputSplit is confusing me .. Any clarifications ??
Posted by maha <ma...@umail.ucsb.edu>.
Sorry I mistyped LineRecordReader by LineInputSplit ... So here is my question again ..
>
> Thanks for the reply .. although I read it has to do with "InputSplit" which represents the data to be processed by an individual Mapper. By default it's a LineRecordReader.
>
> How can I change this property to be FileInputSpilt and Record is the whole File ?
>
> something like JobConf.set ("File.input.format","FileInptSplit");
>
> Is there such way?
>
> Thanks in advance,
> Maha
>
Re: InputSplit is confusing me .. Any clarifications ??
Posted by maha <ma...@umail.ucsb.edu>.
Hi li,
Thanks for the reply .. although I read it has to do with "InputSplit" which represents the data to be processed by an individual Mapper. By default it's a LineInputSplit.
How can I change this property to be FileInputSpilt ?
something like JobConf.set ("File.input.format","FileInptSplit");
Is there such way?
Thanks in advance,
Maha
On Nov 26, 2010, at 9:09 PM, li ping wrote:
> org.apache.hadoop.mapreduce.lib.input.TextInputForma
Re: InputSplit is confusing me .. Any clarifications ??
Posted by li ping <li...@gmail.com>.
That depends on which InputFormat class you use.
For example, The *org.apache.hadoop.mapreduce.lib.input.TextInputFormat. *This
TextInputFormat class will break the file into lines, and the key is the
position of the line. the value is the text of the line.
On Sat, Nov 27, 2010 at 6:08 AM, maha <ma...@umail.ucsb.edu> wrote:
> Hi Everyone,
>
> What I really wish for Thanksgiving is some one giving me clarification
> of how the inputSplit is working.
>
> Eg.
>
> public void map(LongWritable key, Text value, OutputCollector<Text, Text>
> output, Reporter reporter) throws IOException {
> String line = value.toString();
>
> How did we know that map in this case is taking a line and not the whole
> input document ?
>
>
> Happy Thanksgiving everyone,
>
> Maha
--
-----李平