You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "Kimmel, Chad" <ck...@comscore.com> on 2013/07/09 21:32:20 UTC
How do I read headers from first line into schema?
Hi, what I am trying to do is read the headers from the first line as the field names into the schema. For instance, given the following tab deliminated file
--samplefile.txt—
Name Job Age
Chad Engineer 23
Mike Stats 34
Chris IT 25
Instead of deleting the first line and loading in the field names using the AS function:
rows = LOAD 'samplefile.txt’ USING PigStorage('\t') AS (Name:chararray,day,Job:chararray,Age:int);
I would like to instead read it in as part of the PIG script directly. The reason why this is important for my project is because each file being read in has field names which change (i.e. dynamic) for each file, and I need to keep a record of these unique field names.
Does anyone know how to solve this problem? I think the LoadMetaData might be useful, but I don’t know how to use it. Thanks!
Chad
Chad Kimmel Sr. Statistical Analyst | comScore, Inc. (NASDAQ:SCOR)
o +1 (571) 306-6439 | ckimmel@comscore.com<ma...@comscore.com>
...........................................................................................................
comScore Media Metrix® Multi-Platform: Audience Analytics for the Brave New Digital World
www.comscore.com/multiplatform<http://www.comscore.com/multiplatform>
Re: How do I read headers from first line into schema?
Posted by Sajid Raza <wi...@gmail.com>.
Have you considered using a Pig schema?
On Jul 9, 2013, at 12:32 PM, "Kimmel, Chad" <ck...@comscore.com> wrote:
> Hi, what I am trying to do is read the headers from the first line as the field names into the schema. For instance, given the following tab deliminated file
>
> --samplefile.txt—
> Name Job Age
> Chad Engineer 23
> Mike Stats 34
> Chris IT 25
>
> Instead of deleting the first line and loading in the field names using the AS function:
>
> rows = LOAD 'samplefile.txt’ USING PigStorage('\t') AS (Name:chararray,day,Job:chararray,Age:int);
>
> I would like to instead read it in as part of the PIG script directly. The reason why this is important for my project is because each file being read in has field names which change (i.e. dynamic) for each file, and I need to keep a record of these unique field names.
>
> Does anyone know how to solve this problem? I think the LoadMetaData might be useful, but I don’t know how to use it. Thanks!
>
> Chad
>
>
>
> Chad Kimmel Sr. Statistical Analyst | comScore, Inc. (NASDAQ:SCOR)
>
> o +1 (571) 306-6439 | ckimmel@comscore.com<ma...@comscore.com>
>
> ...........................................................................................................
>
> comScore Media Metrix® Multi-Platform: Audience Analytics for the Brave New Digital World
> www.comscore.com/multiplatform<http://www.comscore.com/multiplatform>