You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Yang <te...@gmail.com> on 2012/06/21 05:15:21 UTC
error with FLATTEN and nested generate
I'm trying to run the following code, ---- I had another piece of code that
looks exactly the same , just var name differences, that one runs fine,
but this one gave errors:
2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Apache Pig
version 0.10.0-SNAPSHOT (rexported) compiled Jun 20 2012, 16:31:38
2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Logging error
messages to:
/crypt/yyang_home/work/matching/ML-places-matcher/control_flows/match_suspects/pig_1340248478278.log
2012-06-20 20:14:38,644 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: file:///
2012-06-20 20:14:39,148 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1200: Pig script failed to parse:
<file a.pig, line 11, column 39> Invalid scalar projection: aa : A column
needs to be projected from a relation for it to be used as a scalar
where I'm I doing wrong?
Thanks a lot
yang
######################################
aa = load 'b' as (x:int , y:int);
b = group aa by x;
c = group b by group ;
-- describe c;
-- c: {group: int,b: {(group: int,aa: {(x: int,y: int)})}}
d = foreach c {
xx = foreach b generate FLATTEN(aa) as (o:int, p:int);
generate group , MIN(xx.x);
};
dump d;
Re: error with FLATTEN and nested generate
Posted by Daniel Dai <da...@hortonworks.com>.
Looks like a bug. Change a into $1 works for me.
On Wed, Jun 20, 2012 at 9:15 PM, Yang <te...@gmail.com> wrote:
> thanks, but the error msg is for the line before the xx.x
>
> also I removed the xx.x completely and it still gave errors
>
>
> On Wed, Jun 20, 2012 at 9:09 PM, Bill Graham <bi...@gmail.com> wrote:
>
> > Try replacing MIN(xx.x); with MIN(xx::x);
> >
> > The xx.x syntax returns a scalar IF the xx relationship only has one
> tuple.
> > If it doesn't it fails. The xx::x syntax refers to the x values in the xx
> > relationship.
> >
> > This is a common cause of confusion that we're looking to correct.
> >
> > On Wed, Jun 20, 2012 at 8:15 PM, Yang <te...@gmail.com> wrote:
> >
> > > I'm trying to run the following code, ---- I had another piece of code
> > that
> > > looks exactly the same , just var name differences, that one runs fine,
> > > but this one gave errors:
> > >
> > > 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Apache Pig
> > > version 0.10.0-SNAPSHOT (rexported) compiled Jun 20 2012, 16:31:38
> > > 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Logging
> error
> > > messages to:
> > >
> > >
> >
> /crypt/yyang_home/work/matching/ML-places-matcher/control_flows/match_suspects/pig_1340248478278.log
> > > 2012-06-20 20:14:38,644 [main] INFO
> > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> > > Connecting to hadoop file system at: file:///
> > > 2012-06-20 20:14:39,148 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > > ERROR 1200: Pig script failed to parse:
> > > <file a.pig, line 11, column 39> Invalid scalar projection: aa : A
> column
> > > needs to be projected from a relation for it to be used as a scalar
> > >
> > >
> > >
> > > where I'm I doing wrong?
> > >
> > > Thanks a lot
> > > yang
> > >
> > > ######################################
> > >
> > > aa = load 'b' as (x:int , y:int);
> > >
> > > b = group aa by x;
> > >
> > > c = group b by group ;
> > >
> > > -- describe c;
> > > -- c: {group: int,b: {(group: int,aa: {(x: int,y: int)})}}
> > >
> > > d = foreach c {
> > > xx = foreach b generate FLATTEN(aa) as (o:int, p:int);
> > > generate group , MIN(xx.x);
> > > };
> > >
> > > dump d;
> > >
> >
> >
> >
> > --
> > *Note that I'm no longer using my Yahoo! email address. Please email me
> at
> > billgraham@gmail.com going forward.*
> >
>
Re: error with FLATTEN and nested generate
Posted by Yang <te...@gmail.com>.
thanks, but the error msg is for the line before the xx.x
also I removed the xx.x completely and it still gave errors
On Wed, Jun 20, 2012 at 9:09 PM, Bill Graham <bi...@gmail.com> wrote:
> Try replacing MIN(xx.x); with MIN(xx::x);
>
> The xx.x syntax returns a scalar IF the xx relationship only has one tuple.
> If it doesn't it fails. The xx::x syntax refers to the x values in the xx
> relationship.
>
> This is a common cause of confusion that we're looking to correct.
>
> On Wed, Jun 20, 2012 at 8:15 PM, Yang <te...@gmail.com> wrote:
>
> > I'm trying to run the following code, ---- I had another piece of code
> that
> > looks exactly the same , just var name differences, that one runs fine,
> > but this one gave errors:
> >
> > 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Apache Pig
> > version 0.10.0-SNAPSHOT (rexported) compiled Jun 20 2012, 16:31:38
> > 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Logging error
> > messages to:
> >
> >
> /crypt/yyang_home/work/matching/ML-places-matcher/control_flows/match_suspects/pig_1340248478278.log
> > 2012-06-20 20:14:38,644 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> > Connecting to hadoop file system at: file:///
> > 2012-06-20 20:14:39,148 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 1200: Pig script failed to parse:
> > <file a.pig, line 11, column 39> Invalid scalar projection: aa : A column
> > needs to be projected from a relation for it to be used as a scalar
> >
> >
> >
> > where I'm I doing wrong?
> >
> > Thanks a lot
> > yang
> >
> > ######################################
> >
> > aa = load 'b' as (x:int , y:int);
> >
> > b = group aa by x;
> >
> > c = group b by group ;
> >
> > -- describe c;
> > -- c: {group: int,b: {(group: int,aa: {(x: int,y: int)})}}
> >
> > d = foreach c {
> > xx = foreach b generate FLATTEN(aa) as (o:int, p:int);
> > generate group , MIN(xx.x);
> > };
> >
> > dump d;
> >
>
>
>
> --
> *Note that I'm no longer using my Yahoo! email address. Please email me at
> billgraham@gmail.com going forward.*
>
Re: error with FLATTEN and nested generate
Posted by Bill Graham <bi...@gmail.com>.
Try replacing MIN(xx.x); with MIN(xx::x);
The xx.x syntax returns a scalar IF the xx relationship only has one tuple.
If it doesn't it fails. The xx::x syntax refers to the x values in the xx
relationship.
This is a common cause of confusion that we're looking to correct.
On Wed, Jun 20, 2012 at 8:15 PM, Yang <te...@gmail.com> wrote:
> I'm trying to run the following code, ---- I had another piece of code that
> looks exactly the same , just var name differences, that one runs fine,
> but this one gave errors:
>
> 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0-SNAPSHOT (rexported) compiled Jun 20 2012, 16:31:38
> 2012-06-20 20:14:38,285 [main] INFO org.apache.pig.Main - Logging error
> messages to:
>
> /crypt/yyang_home/work/matching/ML-places-matcher/control_flows/match_suspects/pig_1340248478278.log
> 2012-06-20 20:14:38,644 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting to hadoop file system at: file:///
> 2012-06-20 20:14:39,148 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1200: Pig script failed to parse:
> <file a.pig, line 11, column 39> Invalid scalar projection: aa : A column
> needs to be projected from a relation for it to be used as a scalar
>
>
>
> where I'm I doing wrong?
>
> Thanks a lot
> yang
>
> ######################################
>
> aa = load 'b' as (x:int , y:int);
>
> b = group aa by x;
>
> c = group b by group ;
>
> -- describe c;
> -- c: {group: int,b: {(group: int,aa: {(x: int,y: int)})}}
>
> d = foreach c {
> xx = foreach b generate FLATTEN(aa) as (o:int, p:int);
> generate group , MIN(xx.x);
> };
>
> dump d;
>
--
*Note that I'm no longer using my Yahoo! email address. Please email me at
billgraham@gmail.com going forward.*