Sending the entire file content as value to the mapper

[Kasi Subrahmanyam <ka...@gmail.com>]

Hi Team,

I have a file which has semi structured text  data with no definite start
and end points.
How can i send the entire content of the file at once as key or value to
the mapper instead of line by line.

Thanks,
Subbu

RE: Sending the entire file content as value to the mapper

[Devaraj k <de...@huawei.com>]

Hi,

  You could send the file meta info to the map function as key/value through the split, and then you can read the entire file in your map function.

Thanks
Devaraj k


-----Original Message-----
From: Kasi Subrahmanyam [mailto:kasisubbu440@gmail.com] 
Sent: 11 July 2013 13:38
To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
Subject: Sending the entire file content as value to the mapper

Hi Team,

I have a file which has semi structured text  data with no definite start and end points.
How can i send the entire content of the file at once as key or value to the mapper instead of line by line.

Thanks,
Subbu

RE: Sending the entire file content as value to the mapper

[Charles Baker <cb...@sdl.com>]

Hi Subbu. Sounds like you'll have to implement a custom non-splittable
InputFormat which instantiates a custom RecordReader which in turn consumes
the entire file when it's next(K,V) method is called. Once implemented, you
specify the input format to the JobConf object:

 

conf.setInputFormat(MyInputFormat.class);

 

 

-Chuck

 

From: Kasi Subrahmanyam [mailto:kasisubbu440@gmail.com] 
Sent: Thursday, July 11, 2013 1:08 AM
To: common-user@hadoop.apache.org; mapreduce-user@hadoop.apache.org
Subject: Sending the entire file content as value to the mapper

 

Hi Team,

 

I have a file which has semi structured text  data with no definite start and
end points.

How can i send the entire content of the file at once as key or value to the
mapper instead of line by line.

 

Thanks,

Subbu

</pre><font face="arial" size="2" color="#736F6E">



<a href="http://www.sdl.com/?utm_source=Email&utm_medium=Email%2BSignature&utm_campaign=SDL%2BStandard%2BEmail%2BSignature">
<img src="http://www.sdl.com/email.png" border=0><br><br>www.sdl.com
</a><br><br>

<font face="arial" size="1" color="#736F6E">

<b>SDL PLC confidential, all rights reserved.</b>

If you are not the intended recipient of this mail SDL requests and requires that you delete it without acting upon or copying any of its contents, 
and we further request that you advise us.<BR><BR>
SDL Enterprise Technologies, Inc. - all rights reserved.  The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
<BR>Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
</font>