Help: frustration with types and whatnot while trying to do a conditional sum

[Jonathan Coveney <jc...@gmail.com>]

I appreciate any help you can give. I've searched around and haven't found
anything directly related... I've gone through documentation but can't find
a real reason why this doesn't work.

Here is the jist of my code (n1 is arbitrary, just to group by, n2 is either
null or a large integer):

table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant stuff);
pared = foreach table generate n1, n2;
grouped = group pared by n1;
counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2) ?
0:1))/(double)COUNT(pared.n2) as ratio:double;
ordered = order counted by ratio desc;
limited = limit ordered 200;
dump limited;

This gets this error:

ERROR 1045: Could not infer the matching function for
org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
explicit cast.

If I take out the double parenthesis in the counted sum

ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
chararray,pared: {n1: chararray,n2: chararray}}

I THINK the error is that sum wants the column of a bag as an input, not
actual integers...so I thought I'd try and make that happen by making the
input take the form I want.

So in order to try and get around this, I thought this might work (changing
only these lines)

pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as ooz:int;
grouped = group pared by n1;
counted  = foreach grouped generate group,
(double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;

But this gives this error:
ERROR 1000: Error during parsing. Invalid alias: n2 in {n1: chararray,ooz:
int}

I have no real clue why this fails... I tried breaking it up into two steps
and it doesn't matter.

I'd ideally like to do this without making a UDF, as I feel the base
functionality should support it. Not sure.

Either way, I'd appreciate any help or pointers, as well as any rationale as
to why it does or doesn't work within the pig framework. The whole bag
system is still somewhat counterintuitive.

Thank you for your time

Re: Help: frustration with types and whatnot while trying to do a conditional sum

[Jonathan Coveney <jc...@gmail.com>]

I am sorry, I got confused. I will do that.

2010/11/30 Owen O'Malley <om...@apache.org>

> Pig has moved to its own mailing lists. Please follow up over there.
> -- Owen
>
> On Tue, Nov 30, 2010 at 8:05 AM, Jonathan Coveney <jcoveney@gmail.com
> >wrote:
>
> > I appreciate any help you can give. I've searched around and haven't
> found
> > anything directly related... I've gone through documentation but can't
> find
> > a real reason why this doesn't work.
> >
> > Here is the jist of my code (n1 is arbitrary, just to group by, n2 is
> > either
> > null or a large integer):
> >
> > table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant
> stuff);
> > pared = foreach table generate n1, n2;
> > grouped = group pared by n1;
> > counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2)
> ?
> > 0:1))/(double)COUNT(pared.n2) as ratio:double;
> > ordered = order counted by ratio desc;
> > limited = limit ordered 200;
> > dump limited;
> >
> > This gets this error:
> >
> > ERROR 1045: Could not infer the matching function for
> > org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
> > explicit cast.
> >
> > If I take out the double parenthesis in the counted sum
> >
> > ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
> > chararray,pared: {n1: chararray,n2: chararray}}
> >
> > I THINK the error is that sum wants the column of a bag as an input, not
> > actual integers...so I thought I'd try and make that happen by making the
> > input take the form I want.
> >
> > So in order to try and get around this, I thought this might work
> (changing
> > only these lines)
> >
> > pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as
> ooz:int;
> > grouped = group pared by n1;
> > counted  = foreach grouped generate group,
> > (double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;
> >
> > But this gives this error:
> > ERROR 1000: Error during parsing. Invalid alias: n2 in {n1:
> chararray,ooz:
> > int}
> >
> > I have no real clue why this fails... I tried breaking it up into two
> steps
> > and it doesn't matter.
> >
> > I'd ideally like to do this without making a UDF, as I feel the base
> > functionality should support it. Not sure.
> >
> > Either way, I'd appreciate any help or pointers, as well as any rationale
> > as
> > to why it does or doesn't work within the pig framework. The whole bag
> > system is still somewhat counterintuitive.
> >
> > Thank you for your time
> >
>

Re: Help: frustration with types and whatnot while trying to do a conditional sum

[Owen O'Malley <om...@apache.org>]

Pig has moved to its own mailing lists. Please follow up over there.
-- Owen

On Tue, Nov 30, 2010 at 8:05 AM, Jonathan Coveney <jc...@gmail.com>wrote:

> I appreciate any help you can give. I've searched around and haven't found
> anything directly related... I've gone through documentation but can't find
> a real reason why this doesn't work.
>
> Here is the jist of my code (n1 is arbitrary, just to group by, n2 is
> either
> null or a large integer):
>
> table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant stuff);
> pared = foreach table generate n1, n2;
> grouped = group pared by n1;
> counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2) ?
> 0:1))/(double)COUNT(pared.n2) as ratio:double;
> ordered = order counted by ratio desc;
> limited = limit ordered 200;
> dump limited;
>
> This gets this error:
>
> ERROR 1045: Could not infer the matching function for
> org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
> explicit cast.
>
> If I take out the double parenthesis in the counted sum
>
> ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
> chararray,pared: {n1: chararray,n2: chararray}}
>
> I THINK the error is that sum wants the column of a bag as an input, not
> actual integers...so I thought I'd try and make that happen by making the
> input take the form I want.
>
> So in order to try and get around this, I thought this might work (changing
> only these lines)
>
> pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as ooz:int;
> grouped = group pared by n1;
> counted  = foreach grouped generate group,
> (double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;
>
> But this gives this error:
> ERROR 1000: Error during parsing. Invalid alias: n2 in {n1: chararray,ooz:
> int}
>
> I have no real clue why this fails... I tried breaking it up into two steps
> and it doesn't matter.
>
> I'd ideally like to do this without making a UDF, as I feel the base
> functionality should support it. Not sure.
>
> Either way, I'd appreciate any help or pointers, as well as any rationale
> as
> to why it does or doesn't work within the pig framework. The whole bag
> system is still somewhat counterintuitive.
>
> Thank you for your time
>

Re: Help: frustration with types and whatnot while trying to do a conditional sum

[Owen O'Malley <om...@apache.org>]

Pig has moved to its own mailing lists. Please follow up over there.
-- Owen

On Tue, Nov 30, 2010 at 8:05 AM, Jonathan Coveney <jc...@gmail.com>wrote:

> I appreciate any help you can give. I've searched around and haven't found
> anything directly related... I've gone through documentation but can't find
> a real reason why this doesn't work.
>
> Here is the jist of my code (n1 is arbitrary, just to group by, n2 is
> either
> null or a large integer):
>
> table = LOAD stuff AS (n1:chararray, n2:chararray, other irrelevant stuff);
> pared = foreach table generate n1, n2;
> grouped = group pared by n1;
> counted  = foreach grouped generate group, (double)SUM((IsEmpty(pared.n2) ?
> 0:1))/(double)COUNT(pared.n2) as ratio:double;
> ordered = order counted by ratio desc;
> limited = limit ordered 200;
> dump limited;
>
> This gets this error:
>
> ERROR 1045: Could not infer the matching function for
> org.apache.pig.builtin.SUM as multiple or none of them fit. Please use an
> explicit cast.
>
> If I take out the double parenthesis in the counted sum
>
> ERROR 1000: Error during parsing. Invalid alias: SUM in {group:
> chararray,pared: {n1: chararray,n2: chararray}}
>
> I THINK the error is that sum wants the column of a bag as an input, not
> actual integers...so I thought I'd try and make that happen by making the
> input take the form I want.
>
> So in order to try and get around this, I thought this might work (changing
> only these lines)
>
> pared = foreach beacon_fact generate n1, (IsEmpty(n2) ? 0 : 1) as ooz:int;
> grouped = group pared by n1;
> counted  = foreach grouped generate group,
> (double)SUM(pared.n1)/(double)COUNT(pared.n2) as ratio:double;
>
> But this gives this error:
> ERROR 1000: Error during parsing. Invalid alias: n2 in {n1: chararray,ooz:
> int}
>
> I have no real clue why this fails... I tried breaking it up into two steps
> and it doesn't matter.
>
> I'd ideally like to do this without making a UDF, as I feel the base
> functionality should support it. Not sure.
>
> Either way, I'd appreciate any help or pointers, as well as any rationale
> as
> to why it does or doesn't work within the pig framework. The whole bag
> system is still somewhat counterintuitive.
>
> Thank you for your time
>