You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Robert Yerex <ro...@civitaslearning.com> on 2012/09/18 01:00:12 UTC

Removing unnecessary disambiguation marks

Probably an easy one but...

After processing a file through a series of groupings, aggreagtions and
projections using flatten I end up with long concatenated names for each
field shown in this snippre t from the JsonStorage generated schema

        {

"name"
:"enrollments_instructor_1::enrollments_student_3::enrollments_student_2::enrollments_student_1::enrollments_section::enrollments::term::term_id"
,

            "type":55,

            "description":"autogenerated from Pig Field Schema",

            "schema":null

        },

-- 
Robert Yerex
Data Scientist
Civitas Learning
www.civitaslearning.com

Re: Removing unnecessary disambiguation marks

Posted by Robert Yerex <ro...@civitaslearning.com>.
Thanks! That worked!

On Tue, Sep 18, 2012 at 6:58 AM, Alan Gates <ga...@hortonworks.com> wrote:

> The added foreach will not generate another MR job.
>
> Alan.
>
> On Sep 18, 2012, at 8:54 AM, Ruslan Al-Fakikh wrote:
>
> > Hey,
> >
> > You can try cleaning in a separate FOREACH. I don't think it'll
> > trigger another MR job, but you better check it.
> > Example:
> > resultCleaned = FOREACH result GENERATE
> >                                               name::group::fileldName AS
> fileldName;
> >
> > Ruslan
> >
> > On Tue, Sep 18, 2012 at 3:01 AM, Robert Yerex
> > <ro...@civitaslearning.com> wrote:
> >> Probably an easy one but...
> >>
> >> After processing a file through a series of groupings, aggreagtions and
> >> projections using flatten I end up with long concatenated names for each
> >> field shown in this snippre t from the JsonStorage generated schema
> >>
> >>        {
> >>
> >> "name"
> >>
> :"enrollments_instructor_1::enrollments_student_3::enrollments_student_2::enrollments_student_1::enrollments_section::enrollments::term::term_id"
> >> ,
> >>
> >>            "type":55,
> >>
> >>            "description":"autogenerated from Pig Field Schema",
> >>
> >>            "schema":null
> >>
> >>        },
> >> How do I get rid of all the concatenated naming?
> >>
> >> --
> >> Robert Yerex
> >> Data Scientist
> >> Civitas Learning
> >> www.civitaslearning.com
> >>
> >>
> >>
> >>
> >> --
> >> Robert Yerex
> >> Data Scientist
> >> Civitas Learning
> >> www.civitaslearning.com
>
>


-- 
Robert Yerex
Data Scientist
Civitas Learning
www.civitaslearning.com

Re: Removing unnecessary disambiguation marks

Posted by Alan Gates <ga...@hortonworks.com>.
The added foreach will not generate another MR job.  

Alan.

On Sep 18, 2012, at 8:54 AM, Ruslan Al-Fakikh wrote:

> Hey,
> 
> You can try cleaning in a separate FOREACH. I don't think it'll
> trigger another MR job, but you better check it.
> Example:
> resultCleaned = FOREACH result GENERATE
> 						name::group::fileldName AS fileldName;
> 
> Ruslan
> 
> On Tue, Sep 18, 2012 at 3:01 AM, Robert Yerex
> <ro...@civitaslearning.com> wrote:
>> Probably an easy one but...
>> 
>> After processing a file through a series of groupings, aggreagtions and
>> projections using flatten I end up with long concatenated names for each
>> field shown in this snippre t from the JsonStorage generated schema
>> 
>>        {
>> 
>> "name"
>> :"enrollments_instructor_1::enrollments_student_3::enrollments_student_2::enrollments_student_1::enrollments_section::enrollments::term::term_id"
>> ,
>> 
>>            "type":55,
>> 
>>            "description":"autogenerated from Pig Field Schema",
>> 
>>            "schema":null
>> 
>>        },
>> How do I get rid of all the concatenated naming?
>> 
>> --
>> Robert Yerex
>> Data Scientist
>> Civitas Learning
>> www.civitaslearning.com
>> 
>> 
>> 
>> 
>> --
>> Robert Yerex
>> Data Scientist
>> Civitas Learning
>> www.civitaslearning.com

Re: Removing unnecessary disambiguation marks

Posted by Ruslan Al-Fakikh <me...@gmail.com>.
Hey,

You can try cleaning in a separate FOREACH. I don't think it'll
trigger another MR job, but you better check it.
Example:
resultCleaned = FOREACH result GENERATE
						name::group::fileldName AS fileldName;

Ruslan

On Tue, Sep 18, 2012 at 3:01 AM, Robert Yerex
<ro...@civitaslearning.com> wrote:
> Probably an easy one but...
>
> After processing a file through a series of groupings, aggreagtions and
> projections using flatten I end up with long concatenated names for each
> field shown in this snippre t from the JsonStorage generated schema
>
>         {
>
> "name"
> :"enrollments_instructor_1::enrollments_student_3::enrollments_student_2::enrollments_student_1::enrollments_section::enrollments::term::term_id"
> ,
>
>             "type":55,
>
>             "description":"autogenerated from Pig Field Schema",
>
>             "schema":null
>
>         },
> How do I get rid of all the concatenated naming?
>
> --
> Robert Yerex
> Data Scientist
> Civitas Learning
> www.civitaslearning.com
>
>
>
>
> --
> Robert Yerex
> Data Scientist
> Civitas Learning
> www.civitaslearning.com

Fwd: Removing unnecessary disambiguation marks

Posted by Robert Yerex <ro...@civitaslearning.com>.
Probably an easy one but...

After processing a file through a series of groupings, aggreagtions and
projections using flatten I end up with long concatenated names for each
field shown in this snippre t from the JsonStorage generated schema

        {

"name"
:"enrollments_instructor_1::enrollments_student_3::enrollments_student_2::enrollments_student_1::enrollments_section::enrollments::term::term_id"
,

            "type":55,

            "description":"autogenerated from Pig Field Schema",

            "schema":null

        },
How do I get rid of all the concatenated naming?

-- 
Robert Yerex
Data Scientist
Civitas Learning
www.civitaslearning.com




-- 
Robert Yerex
Data Scientist
Civitas Learning
www.civitaslearning.com