You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by JAGANADH G <ja...@gmail.com> on 2012/10/03 18:49:10 UTC
Loading text file
Hi All
Is there any way to load a text file as single record (text:chararray) in
Pig.
I am trying to load a bunch of text files from a directory . But it keeps
each line as single record.
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in
Re: Loading text file
Posted by Alan Gates <ga...@hortonworks.com>.
There is not a pre-built load function to do that. In fact I am not aware of a Hadoop InputFormat that does that. So you would first need to subclass Hadoop's FileInputFormat and then write a Load Func. Both should be fairly straight forward since all you need to do is remove the record and field parsing from existing code.
Alan.
On Oct 3, 2012, at 9:49 AM, JAGANADH G wrote:
> Hi All
>
> Is there any way to load a text file as single record (text:chararray) in
> Pig.
>
> I am trying to load a bunch of text files from a directory . But it keeps
> each line as single record.
>
>
> --
> **********************************
> JAGANADH G
> http://jaganadhg.in
> *ILUGCBE*
> http://ilugcbe.org.in
Re: Loading text file
Posted by Jonathan Coveney <jc...@gmail.com>.
Here is an InputFormat that does this:
https://gist.github.com/808035
2012/10/3 Siddharth Tiwari <si...@tcs.com>
> You will have to write your own loader. You will have to modify the
> InputFormat which uses \n \r or EOF as split delimiter.
>
>
>
>
>
> -----JAGANADH G <ja...@gmail.com> wrote: -----
> To: user@pig.apache.org
> From: JAGANADH G <ja...@gmail.com>
> Date: 10/03/2012 10:20PM
> Subject: Loading text file
>
> Hi All
>
> Is there any way to load a text file as single record (text:chararray) in
> Pig.
>
> I am trying to load a bunch of text files from a directory . But it keeps
> each line as single record.
>
>
> --
> **********************************
> JAGANADH G
> http://jaganadhg.in
> *ILUGCBE*
> http://ilugcbe.org.in
> =====-----=====-----=====
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>
>
Re: Loading text file
Posted by Siddharth Tiwari <si...@tcs.com>.
You will have to write your own loader. You will have to modify the InputFormat which uses \n \r or EOF as split delimiter.
-----JAGANADH G <ja...@gmail.com> wrote: -----
To: user@pig.apache.org
From: JAGANADH G <ja...@gmail.com>
Date: 10/03/2012 10:20PM
Subject: Loading text file
Hi All
Is there any way to load a text file as single record (text:chararray) in
Pig.
I am trying to load a bunch of text files from a directory . But it keeps
each line as single record.
--
**********************************
JAGANADH G
http://jaganadhg.in
*ILUGCBE*
http://ilugcbe.org.in
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you