You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Alan Gates <ga...@yahoo-inc.com> on 2008/06/05 18:10:45 UTC
Issues with group as an alias
Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
(or set of fields) that are grouped on are given the alias 'group'.
This has a couple of issues:
1) It's confusing. 'group' is now a keyword and an alias.
2) We don't currently allow 'group' as an alias in an AS. It is
strange to have an alias that can only be assigned by the language and
never by the user.
Possible solutions:
I) Status quo. We could fix it so that group is allowed to be assigned
as an alias in AS.
Pros: Backward compatibility
Cons: a) will make the parser more complicated
b) see 1) above.
II) Don't give an implicit alias to the group key(s). If users want an
alias, they can assign it using AS.
Pros: Simplicity
Cons: We do assign aliases to grouped bags. That is, if we have C =
GROUP B by $0 the resulting schema of C is (group, B). So if we don't
assign an alias to the group key, we now have a schema ($0, B). This
seems strange. And worse yet, if users want to alias the group key(s),
they'll be forced to alias all the grouped bags as well.
III) Carry the alias (if any) that the field had before. So if we had a
script like:
A = load 'myfile' as (x, y, z);
B = group A by x;
The the schema of B would be (x, A). This is quite natural for grouping
of single columns. But it turns nasty when you group on multiple
columns. Do we then append the names to together? So if you have
B = group A by x, y;
is the resulting schema (x_y, A)? Ugh.
In this case there is also the question of what to do in the case of
cogroups, where the key may be named differently in different relations.
A = load 'myfile' as (x, y, z);
B = load 'myotherfile' as (t, u, v);
C = cogroup A by x, B by t;
Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
could be resolved by either saying first one always wins, or allowing
either.
Pros: Very natural for the users, their fields maintain names through
the query.
Cons: Quickly gets burdensome in the case of multi-key groups.
IV) Assign a non-keyword alias to the group key, like grp or groupkey or
grpkey (or some other suitable choice).
Pros: Least disruptive change. Users only have to go through their
scripts and find places where they use the group alias and change it to
grp (or whatever).
Cons: Still leaves us with a situation where we are assigning a name to
a field arbtrarily, leaving users confused as to how their fields got
named that.
V) Remove GROUP as a keyword. It is just short for COGROUP of one
relation anyway.
Pros: Smaller syntax in a language is always good.
Cons: Will break a lot of scripts, and confuse a lot of users who only
think in terms of GROUP and JOIN and never use COGROUP explicitly.
One could also conceive of combinations of these. For example, we
always assign a name like grpkey to the group key(s), and in the single
key case we also carry forward the alias that the field already had, if any.
Thoughts? Other possibilities?
Alan.
Re: Issues with group as an alias
Posted by Alan Gates <ga...@yahoo-inc.com>.
After 2+ weeks I think it's time to resolve this issue. At this point
we have 2 votes for option I (Ben, Pi), and 4 votes for option III
(Alan, Olga, Chris, Ted Dunning). So as part of the pipeline rework
we'll make this change.
Alan.
Alan Gates wrote:
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the
> field (or set of fields) that are grouped on are given the alias
> 'group'. This has a couple of issues:
>
> 1) It's confusing. 'group' is now a keyword and an alias.
> 2) We don't currently allow 'group' as an alias in an AS. It is
> strange to have an alias that can only be assigned by the language and
> never by the user.
>
> Possible solutions:
>
> I) Status quo. We could fix it so that group is allowed to be
> assigned as an alias in AS.
>
> Pros: Backward compatibility
> Cons: a) will make the parser more complicated
> b) see 1) above.
>
>
> II) Don't give an implicit alias to the group key(s). If users want
> an alias, they can assign it using AS.
>
> Pros: Simplicity
> Cons: We do assign aliases to grouped bags. That is, if we have C =
> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> assign an alias to the group key, we now have a schema ($0, B). This
> seems strange. And worse yet, if users want to alias the group
> key(s), they'll be forced to alias all the grouped bags as well.
>
> III) Carry the alias (if any) that the field had before. So if we had
> a script like:
>
> A = load 'myfile' as (x, y, z);
> B = group A by x;
>
> The the schema of B would be (x, A). This is quite natural for
> grouping of single columns. But it turns nasty when you group on
> multiple columns. Do we then append the names to together? So if you
> have
>
> B = group A by x, y;
>
> is the resulting schema (x_y, A)? Ugh.
>
> In this case there is also the question of what to do in the case of
> cogroups, where the key may be named differently in different relations.
>
> A = load 'myfile' as (x, y, z);
> B = load 'myotherfile' as (t, u, v);
> C = cogroup A by x, B by t;
>
> Is the resulting schema (x, A, B) or (t, A, B) or are both valid?
> This could be resolved by either saying first one always wins, or
> allowing either.
>
> Pros: Very natural for the users, their fields maintain names through
> the query.
> Cons: Quickly gets burdensome in the case of multi-key groups.
>
> IV) Assign a non-keyword alias to the group key, like grp or groupkey
> or grpkey (or some other suitable choice).
> Pros: Least disruptive change. Users only have to go through their
> scripts and find places where they use the group alias and change it
> to grp (or whatever).
> Cons: Still leaves us with a situation where we are assigning a name
> to a field arbtrarily, leaving users confused as to how their fields
> got named that.
>
> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> relation anyway.
>
> Pros: Smaller syntax in a language is always good.
> Cons: Will break a lot of scripts, and confuse a lot of users who
> only think in terms of GROUP and JOIN and never use COGROUP explicitly.
>
> One could also conceive of combinations of these. For example, we
> always assign a name like grpkey to the group key(s), and in the
> single key case we also carry forward the alias that the field already
> had, if any.
>
> Thoughts? Other possibilities?
>
> Alan.
Re: Issues with group as an alias
Posted by pi song <pi...@gmail.com>.
I think it is not totally orthogonal as the answer to my question can be
used to eliminate option (I).
I agree on option (III) due to usability reason.
Pi
On 6/10/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>
> The issue of non-reserved keywords is orthogonal to the issue at hand: what
> is the most natural way to name the group key (i.e., we can still allow
> non-reserved keywords, and select a different way of naming the group key,
> if we want).
>
> Every time I give a tutorial on Pig, people struggle to understand what
> this mysterious "group" field is. It is ugly and non-intuitive.
>
> Option III is far more natural, and will cover 95% of the cases (for the
> rest of the cases, the user is doing something complicated so I think it's
> okay for them to name the group key manually).
>
> -Chris
>
>
> On Jun 9, 2008, at 3:39 AM, pi song wrote:
>
> I prefer (I) and that means I want to allow non-reserved keywords.
>>
>> On Fri, Jun 6, 2008 at 9:33 AM, pi song <pi...@gmail.com> wrote:
>>
>> I know it is very subjective to say I don't agree with "1) It's
>>> confusing". On developers' side, it is. But on users' side, it might not.
>>>
>>> Some languages allow usage of keywords given they are used in the right
>>> context. The current Pig implementation also allows referring to "group"
>>> as
>>> an alias.
>>>
>>> Before we jump to the solution, shouldn't it be better to make our
>>> position clear on "Do we want every keyword to be reserved word
>>> regardless
>>> of context?"
>>>
>>> Pi
>>>
>>>
>>> On 6/6/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>>>
>>>>
>>>> I vote for (III) -- propagate the alias. This makes the scripts very
>>>> natural and readable, e.g.:
>>>>
>>>> a = group pages by host;
>>>> b = foreach a generate host, count(pages);
>>>>
>>>> As for what to do in the case of grouping on multiple fields, or
>>>> co-group
>>>> on differently-named fields, we should *not* assign a default name --
>>>> the
>>>> user can choose a name using "AS".
>>>>
>>>> -Chris
>>>>
>>>>
>>>> On Jun 5, 2008, at 9:10 AM, Alan Gates wrote:
>>>>
>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
>>>>
>>>>> (or set of fields) that are grouped on are given the alias 'group'.
>>>>> This
>>>>> has a couple of issues:
>>>>>
>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>> 2) We don't currently allow 'group' as an alias in an AS. It is
>>>>> strange
>>>>> to have an alias that can only be assigned by the language and never by
>>>>> the
>>>>> user.
>>>>>
>>>>> Possible solutions:
>>>>>
>>>>> I) Status quo. We could fix it so that group is allowed to be assigned
>>>>> as an alias in AS.
>>>>>
>>>>> Pros: Backward compatibility
>>>>> Cons: a) will make the parser more complicated
>>>>> b) see 1) above.
>>>>>
>>>>>
>>>>> II) Don't give an implicit alias to the group key(s). If users want an
>>>>> alias, they can assign it using AS.
>>>>>
>>>>> Pros: Simplicity
>>>>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>>>>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>>>>> assign an alias to the group key, we now have a schema ($0, B). This
>>>>> seems
>>>>> strange. And worse yet, if users want to alias the group key(s),
>>>>> they'll be
>>>>> forced to alias all the grouped bags as well.
>>>>>
>>>>> III) Carry the alias (if any) that the field had before. So if we had
>>>>> a
>>>>> script like:
>>>>>
>>>>> A = load 'myfile' as (x, y, z);
>>>>> B = group A by x;
>>>>>
>>>>> The the schema of B would be (x, A). This is quite natural for
>>>>> grouping
>>>>> of single columns. But it turns nasty when you group on multiple
>>>>> columns.
>>>>> Do we then append the names to together? So if you have
>>>>>
>>>>> B = group A by x, y;
>>>>>
>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>
>>>>> In this case there is also the question of what to do in the case of
>>>>> cogroups, where the key may be named differently in different
>>>>> relations.
>>>>>
>>>>> A = load 'myfile' as (x, y, z);
>>>>> B = load 'myotherfile' as (t, u, v);
>>>>> C = cogroup A by x, B by t;
>>>>>
>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
>>>>> could be resolved by either saying first one always wins, or allowing
>>>>> either.
>>>>>
>>>>> Pros: Very natural for the users, their fields maintain names through
>>>>> the query.
>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>>
>>>>> IV) Assign a non-keyword alias to the group key, like grp or groupkey
>>>>> or
>>>>> grpkey (or some other suitable choice).
>>>>> Pros: Least disruptive change. Users only have to go through their
>>>>> scripts and find places where they use the group alias and change it to
>>>>> grp
>>>>> (or whatever).
>>>>> Cons: Still leaves us with a situation where we are assigning a name
>>>>> to
>>>>> a field arbtrarily, leaving users confused as to how their fields got
>>>>> named
>>>>> that.
>>>>>
>>>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>>>> relation anyway.
>>>>>
>>>>> Pros: Smaller syntax in a language is always good.
>>>>> Cons: Will break a lot of scripts, and confuse a lot of users who only
>>>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>>>
>>>>> One could also conceive of combinations of these. For example, we
>>>>> always
>>>>> assign a name like grpkey to the group key(s), and in the single key
>>>>> case we
>>>>> also carry forward the alias that the field already had, if any.
>>>>>
>>>>> Thoughts? Other possibilities?
>>>>>
>>>>> Alan.
>>>>>
>>>>>
>>>> --
>>>> Christopher Olston, Ph.D.
>>>> Sr. Research Scientist
>>>> Yahoo! Research
>>>>
>>>>
>>>>
>>>>
>>>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
The issue of non-reserved keywords is orthogonal to the issue at
hand: what is the most natural way to name the group key (i.e., we
can still allow non-reserved keywords, and select a different way of
naming the group key, if we want).
Every time I give a tutorial on Pig, people struggle to understand
what this mysterious "group" field is. It is ugly and non-intuitive.
Option III is far more natural, and will cover 95% of the cases (for
the rest of the cases, the user is doing something complicated so I
think it's okay for them to name the group key manually).
-Chris
On Jun 9, 2008, at 3:39 AM, pi song wrote:
> I prefer (I) and that means I want to allow non-reserved keywords.
>
> On Fri, Jun 6, 2008 at 9:33 AM, pi song <pi...@gmail.com> wrote:
>
>> I know it is very subjective to say I don't agree with "1) It's
>> confusing". On developers' side, it is. But on users' side, it
>> might not.
>>
>> Some languages allow usage of keywords given they are used in the
>> right
>> context. The current Pig implementation also allows referring to
>> "group" as
>> an alias.
>>
>> Before we jump to the solution, shouldn't it be better to make our
>> position clear on "Do we want every keyword to be reserved word
>> regardless
>> of context?"
>>
>> Pi
>>
>>
>> On 6/6/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>>>
>>> I vote for (III) -- propagate the alias. This makes the scripts very
>>> natural and readable, e.g.:
>>>
>>> a = group pages by host;
>>> b = foreach a generate host, count(pages);
>>>
>>> As for what to do in the case of grouping on multiple fields, or
>>> co-group
>>> on differently-named fields, we should *not* assign a default
>>> name -- the
>>> user can choose a name using "AS".
>>>
>>> -Chris
>>>
>>>
>>> On Jun 5, 2008, at 9:10 AM, Alan Gates wrote:
>>>
>>> Currently in Pig Latin, anytime a (CO)GROUP statement is used,
>>> the field
>>>> (or set of fields) that are grouped on are given the alias
>>>> 'group'. This
>>>> has a couple of issues:
>>>>
>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>> 2) We don't currently allow 'group' as an alias in an AS. It
>>>> is strange
>>>> to have an alias that can only be assigned by the language and
>>>> never by the
>>>> user.
>>>>
>>>> Possible solutions:
>>>>
>>>> I) Status quo. We could fix it so that group is allowed to be
>>>> assigned
>>>> as an alias in AS.
>>>>
>>>> Pros: Backward compatibility
>>>> Cons: a) will make the parser more complicated
>>>> b) see 1) above.
>>>>
>>>>
>>>> II) Don't give an implicit alias to the group key(s). If users
>>>> want an
>>>> alias, they can assign it using AS.
>>>>
>>>> Pros: Simplicity
>>>> Cons: We do assign aliases to grouped bags. That is, if we
>>>> have C =
>>>> GROUP B by $0 the resulting schema of C is (group, B). So if we
>>>> don't
>>>> assign an alias to the group key, we now have a schema ($0, B).
>>>> This seems
>>>> strange. And worse yet, if users want to alias the group key
>>>> (s), they'll be
>>>> forced to alias all the grouped bags as well.
>>>>
>>>> III) Carry the alias (if any) that the field had before. So if
>>>> we had a
>>>> script like:
>>>>
>>>> A = load 'myfile' as (x, y, z);
>>>> B = group A by x;
>>>>
>>>> The the schema of B would be (x, A). This is quite natural for
>>>> grouping
>>>> of single columns. But it turns nasty when you group on
>>>> multiple columns.
>>>> Do we then append the names to together? So if you have
>>>>
>>>> B = group A by x, y;
>>>>
>>>> is the resulting schema (x_y, A)? Ugh.
>>>>
>>>> In this case there is also the question of what to do in the
>>>> case of
>>>> cogroups, where the key may be named differently in different
>>>> relations.
>>>>
>>>> A = load 'myfile' as (x, y, z);
>>>> B = load 'myotherfile' as (t, u, v);
>>>> C = cogroup A by x, B by t;
>>>>
>>>> Is the resulting schema (x, A, B) or (t, A, B) or are both
>>>> valid? This
>>>> could be resolved by either saying first one always wins, or
>>>> allowing
>>>> either.
>>>>
>>>> Pros: Very natural for the users, their fields maintain names
>>>> through
>>>> the query.
>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>
>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>> groupkey or
>>>> grpkey (or some other suitable choice).
>>>> Pros: Least disruptive change. Users only have to go through
>>>> their
>>>> scripts and find places where they use the group alias and
>>>> change it to grp
>>>> (or whatever).
>>>> Cons: Still leaves us with a situation where we are assigning a
>>>> name to
>>>> a field arbtrarily, leaving users confused as to how their
>>>> fields got named
>>>> that.
>>>>
>>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>>> relation anyway.
>>>>
>>>> Pros: Smaller syntax in a language is always good.
>>>> Cons: Will break a lot of scripts, and confuse a lot of users
>>>> who only
>>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>>
>>>> One could also conceive of combinations of these. For example,
>>>> we always
>>>> assign a name like grpkey to the group key(s), and in the single
>>>> key case we
>>>> also carry forward the alias that the field already had, if any.
>>>>
>>>> Thoughts? Other possibilities?
>>>>
>>>> Alan.
>>>>
>>>
>>> --
>>> Christopher Olston, Ph.D.
>>> Sr. Research Scientist
>>> Yahoo! Research
>>>
>>>
>>>
>>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
Re: Issues with group as an alias
Posted by pi song <pi...@gmail.com>.
I prefer (I) and that means I want to allow non-reserved keywords.
On Fri, Jun 6, 2008 at 9:33 AM, pi song <pi...@gmail.com> wrote:
> I know it is very subjective to say I don't agree with "1) It's
> confusing". On developers' side, it is. But on users' side, it might not.
>
> Some languages allow usage of keywords given they are used in the right
> context. The current Pig implementation also allows referring to "group" as
> an alias.
>
> Before we jump to the solution, shouldn't it be better to make our
> position clear on "Do we want every keyword to be reserved word regardless
> of context?"
>
> Pi
>
>
> On 6/6/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>>
>> I vote for (III) -- propagate the alias. This makes the scripts very
>> natural and readable, e.g.:
>>
>> a = group pages by host;
>> b = foreach a generate host, count(pages);
>>
>> As for what to do in the case of grouping on multiple fields, or co-group
>> on differently-named fields, we should *not* assign a default name -- the
>> user can choose a name using "AS".
>>
>> -Chris
>>
>>
>> On Jun 5, 2008, at 9:10 AM, Alan Gates wrote:
>>
>> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
>>> (or set of fields) that are grouped on are given the alias 'group'. This
>>> has a couple of issues:
>>>
>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>> 2) We don't currently allow 'group' as an alias in an AS. It is strange
>>> to have an alias that can only be assigned by the language and never by the
>>> user.
>>>
>>> Possible solutions:
>>>
>>> I) Status quo. We could fix it so that group is allowed to be assigned
>>> as an alias in AS.
>>>
>>> Pros: Backward compatibility
>>> Cons: a) will make the parser more complicated
>>> b) see 1) above.
>>>
>>>
>>> II) Don't give an implicit alias to the group key(s). If users want an
>>> alias, they can assign it using AS.
>>>
>>> Pros: Simplicity
>>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>>> assign an alias to the group key, we now have a schema ($0, B). This seems
>>> strange. And worse yet, if users want to alias the group key(s), they'll be
>>> forced to alias all the grouped bags as well.
>>>
>>> III) Carry the alias (if any) that the field had before. So if we had a
>>> script like:
>>>
>>> A = load 'myfile' as (x, y, z);
>>> B = group A by x;
>>>
>>> The the schema of B would be (x, A). This is quite natural for grouping
>>> of single columns. But it turns nasty when you group on multiple columns.
>>> Do we then append the names to together? So if you have
>>>
>>> B = group A by x, y;
>>>
>>> is the resulting schema (x_y, A)? Ugh.
>>>
>>> In this case there is also the question of what to do in the case of
>>> cogroups, where the key may be named differently in different relations.
>>>
>>> A = load 'myfile' as (x, y, z);
>>> B = load 'myotherfile' as (t, u, v);
>>> C = cogroup A by x, B by t;
>>>
>>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
>>> could be resolved by either saying first one always wins, or allowing
>>> either.
>>>
>>> Pros: Very natural for the users, their fields maintain names through
>>> the query.
>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>
>>> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
>>> grpkey (or some other suitable choice).
>>> Pros: Least disruptive change. Users only have to go through their
>>> scripts and find places where they use the group alias and change it to grp
>>> (or whatever).
>>> Cons: Still leaves us with a situation where we are assigning a name to
>>> a field arbtrarily, leaving users confused as to how their fields got named
>>> that.
>>>
>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>> relation anyway.
>>>
>>> Pros: Smaller syntax in a language is always good.
>>> Cons: Will break a lot of scripts, and confuse a lot of users who only
>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>
>>> One could also conceive of combinations of these. For example, we always
>>> assign a name like grpkey to the group key(s), and in the single key case we
>>> also carry forward the alias that the field already had, if any.
>>>
>>> Thoughts? Other possibilities?
>>>
>>> Alan.
>>>
>>
>> --
>> Christopher Olston, Ph.D.
>> Sr. Research Scientist
>> Yahoo! Research
>>
>>
>>
>
Re: Issues with group as an alias
Posted by pi song <pi...@gmail.com>.
I know it is very subjective to say I don't agree with "1) It's confusing".
On developers' side, it is. But on users' side, it might not.
Some languages allow usage of keywords given they are used in the right
context. The current Pig implementation also allows referring to "group" as
an alias.
Before we jump to the solution, shouldn't it be better to make our
position clear on "Do we want every keyword to be reserved word regardless
of context?"
Pi
On 6/6/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>
> I vote for (III) -- propagate the alias. This makes the scripts very
> natural and readable, e.g.:
>
> a = group pages by host;
> b = foreach a generate host, count(pages);
>
> As for what to do in the case of grouping on multiple fields, or co-group
> on differently-named fields, we should *not* assign a default name -- the
> user can choose a name using "AS".
>
> -Chris
>
>
> On Jun 5, 2008, at 9:10 AM, Alan Gates wrote:
>
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
>> (or set of fields) that are grouped on are given the alias 'group'. This
>> has a couple of issues:
>>
>> 1) It's confusing. 'group' is now a keyword and an alias.
>> 2) We don't currently allow 'group' as an alias in an AS. It is strange
>> to have an alias that can only be assigned by the language and never by the
>> user.
>>
>> Possible solutions:
>>
>> I) Status quo. We could fix it so that group is allowed to be assigned as
>> an alias in AS.
>>
>> Pros: Backward compatibility
>> Cons: a) will make the parser more complicated
>> b) see 1) above.
>>
>>
>> II) Don't give an implicit alias to the group key(s). If users want an
>> alias, they can assign it using AS.
>>
>> Pros: Simplicity
>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>> assign an alias to the group key, we now have a schema ($0, B). This seems
>> strange. And worse yet, if users want to alias the group key(s), they'll be
>> forced to alias all the grouped bags as well.
>>
>> III) Carry the alias (if any) that the field had before. So if we had a
>> script like:
>>
>> A = load 'myfile' as (x, y, z);
>> B = group A by x;
>>
>> The the schema of B would be (x, A). This is quite natural for grouping
>> of single columns. But it turns nasty when you group on multiple columns.
>> Do we then append the names to together? So if you have
>>
>> B = group A by x, y;
>>
>> is the resulting schema (x_y, A)? Ugh.
>>
>> In this case there is also the question of what to do in the case of
>> cogroups, where the key may be named differently in different relations.
>>
>> A = load 'myfile' as (x, y, z);
>> B = load 'myotherfile' as (t, u, v);
>> C = cogroup A by x, B by t;
>>
>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
>> could be resolved by either saying first one always wins, or allowing
>> either.
>>
>> Pros: Very natural for the users, their fields maintain names through the
>> query.
>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>
>> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
>> grpkey (or some other suitable choice).
>> Pros: Least disruptive change. Users only have to go through their
>> scripts and find places where they use the group alias and change it to grp
>> (or whatever).
>> Cons: Still leaves us with a situation where we are assigning a name to a
>> field arbtrarily, leaving users confused as to how their fields got named
>> that.
>>
>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>> relation anyway.
>>
>> Pros: Smaller syntax in a language is always good.
>> Cons: Will break a lot of scripts, and confuse a lot of users who only
>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>
>> One could also conceive of combinations of these. For example, we always
>> assign a name like grpkey to the group key(s), and in the single key case we
>> also carry forward the alias that the field already had, if any.
>>
>> Thoughts? Other possibilities?
>>
>> Alan.
>>
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
I vote for (III) -- propagate the alias. This makes the scripts very
natural and readable, e.g.:
a = group pages by host;
b = foreach a generate host, count(pages);
As for what to do in the case of grouping on multiple fields, or co-
group on differently-named fields, we should *not* assign a default
name -- the user can choose a name using "AS".
-Chris
On Jun 5, 2008, at 9:10 AM, Alan Gates wrote:
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the
> field (or set of fields) that are grouped on are given the alias
> 'group'. This has a couple of issues:
>
> 1) It's confusing. 'group' is now a keyword and an alias.
> 2) We don't currently allow 'group' as an alias in an AS. It is
> strange to have an alias that can only be assigned by the language
> and never by the user.
>
> Possible solutions:
>
> I) Status quo. We could fix it so that group is allowed to be
> assigned as an alias in AS.
>
> Pros: Backward compatibility
> Cons: a) will make the parser more complicated
> b) see 1) above.
>
>
> II) Don't give an implicit alias to the group key(s). If users
> want an alias, they can assign it using AS.
>
> Pros: Simplicity
> Cons: We do assign aliases to grouped bags. That is, if we have C
> = GROUP B by $0 the resulting schema of C is (group, B). So if we
> don't assign an alias to the group key, we now have a schema ($0,
> B). This seems strange. And worse yet, if users want to alias the
> group key(s), they'll be forced to alias all the grouped bags as well.
>
> III) Carry the alias (if any) that the field had before. So if we
> had a script like:
>
> A = load 'myfile' as (x, y, z);
> B = group A by x;
>
> The the schema of B would be (x, A). This is quite natural for
> grouping of single columns. But it turns nasty when you group on
> multiple columns. Do we then append the names to together? So if
> you have
>
> B = group A by x, y;
>
> is the resulting schema (x_y, A)? Ugh.
>
> In this case there is also the question of what to do in the case
> of cogroups, where the key may be named differently in different
> relations.
>
> A = load 'myfile' as (x, y, z);
> B = load 'myotherfile' as (t, u, v);
> C = cogroup A by x, B by t;
>
> Is the resulting schema (x, A, B) or (t, A, B) or are both valid?
> This could be resolved by either saying first one always wins, or
> allowing either.
>
> Pros: Very natural for the users, their fields maintain names
> through the query.
> Cons: Quickly gets burdensome in the case of multi-key groups.
>
> IV) Assign a non-keyword alias to the group key, like grp or
> groupkey or grpkey (or some other suitable choice).
> Pros: Least disruptive change. Users only have to go through
> their scripts and find places where they use the group alias and
> change it to grp (or whatever).
> Cons: Still leaves us with a situation where we are assigning a
> name to a field arbtrarily, leaving users confused as to how their
> fields got named that.
>
> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> relation anyway.
>
> Pros: Smaller syntax in a language is always good.
> Cons: Will break a lot of scripts, and confuse a lot of users who
> only think in terms of GROUP and JOIN and never use COGROUP
> explicitly.
>
> One could also conceive of combinations of these. For example, we
> always assign a name like grpkey to the group key(s), and in the
> single key case we also carry forward the alias that the field
> already had, if any.
>
> Thoughts? Other possibilities?
>
> Alan.
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
Re: Issues with group as an alias
Posted by Benjamin Reed <br...@yahoo-inc.com>.
I prefer option I). I agree that "group" may not be the optimal choice for the
alias name, but it's not that bad and I would opt for being less disruptive.
As far as the parser change, I think it is just changing:
<IDENTIFIER: (<LETTER>)+(<DIGIT>|<LETTER>|<SPECIALCHAR>|"::")*>
to
<IDENTIFIER: (<LETTER>)+(<DIGIT>|<LETTER>|<SPECIALCHAR>|"::")*|<GROUP>>
It might be good to open up a Jira so that we can actually record votes.
ben
On Thursday 05 June 2008 09:10:45 Alan Gates wrote:
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
> (or set of fields) that are grouped on are given the alias 'group'.
> This has a couple of issues:
>
> 1) It's confusing. 'group' is now a keyword and an alias.
> 2) We don't currently allow 'group' as an alias in an AS. It is
> strange to have an alias that can only be assigned by the language and
> never by the user.
>
> Possible solutions:
>
> I) Status quo. We could fix it so that group is allowed to be assigned
> as an alias in AS.
>
> Pros: Backward compatibility
> Cons: a) will make the parser more complicated
> b) see 1) above.
>
>
> II) Don't give an implicit alias to the group key(s). If users want an
> alias, they can assign it using AS.
>
> Pros: Simplicity
> Cons: We do assign aliases to grouped bags. That is, if we have C =
> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> assign an alias to the group key, we now have a schema ($0, B). This
> seems strange. And worse yet, if users want to alias the group key(s),
> they'll be forced to alias all the grouped bags as well.
>
> III) Carry the alias (if any) that the field had before. So if we had a
> script like:
>
> A = load 'myfile' as (x, y, z);
> B = group A by x;
>
> The the schema of B would be (x, A). This is quite natural for grouping
> of single columns. But it turns nasty when you group on multiple
> columns. Do we then append the names to together? So if you have
>
> B = group A by x, y;
>
> is the resulting schema (x_y, A)? Ugh.
>
> In this case there is also the question of what to do in the case of
> cogroups, where the key may be named differently in different relations.
>
> A = load 'myfile' as (x, y, z);
> B = load 'myotherfile' as (t, u, v);
> C = cogroup A by x, B by t;
>
> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
> could be resolved by either saying first one always wins, or allowing
> either.
>
> Pros: Very natural for the users, their fields maintain names through
> the query.
> Cons: Quickly gets burdensome in the case of multi-key groups.
>
> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
> grpkey (or some other suitable choice).
>
> Pros: Least disruptive change. Users only have to go through their
> scripts and find places where they use the group alias and change it to
> grp (or whatever).
> Cons: Still leaves us with a situation where we are assigning a name to
> a field arbtrarily, leaving users confused as to how their fields got
> named that.
>
> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> relation anyway.
>
> Pros: Smaller syntax in a language is always good.
> Cons: Will break a lot of scripts, and confuse a lot of users who only
> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>
> One could also conceive of combinations of these. For example, we
> always assign a name like grpkey to the group key(s), and in the single
> key case we also carry forward the alias that the field already had, if
> any.
>
> Thoughts? Other possibilities?
>
> Alan.
RE: Issues with group as an alias
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
I am fine with 3. Can we in addition allow access by position as well as
let users assign their own names?
Olga
> -----Original Message-----
> From: Alan Gates [mailto:gates@yahoo-inc.com]
> Sent: Friday, June 13, 2008 7:26 AM
> To: pig-dev@incubator.apache.org
> Subject: Re: Issues with group as an alias
>
> All,
>
> I too will vote for III, with the caveat that we don't give
> names to multi-field grouping keys. We need to make sure we
> support AS to allow the user to name their grouping keys if they want.
>
> So far, the vote totals are:
> I: 1
> II: 0
> III: 3
> IV: 0
> V: 0
>
> I'd like to make a decision and move forward by mid next
> week. If you haven't voted and you'd like to, please do so
> now. If you feel passionately about one of the options that
> is loosing, please make your arguments now.
>
> Alan.
>
> Alan Gates wrote:
> > Currently in Pig Latin, anytime a (CO)GROUP statement is used, the
> > field (or set of fields) that are grouped on are given the alias
> > 'group'. This has a couple of issues:
> >
> > 1) It's confusing. 'group' is now a keyword and an alias.
> > 2) We don't currently allow 'group' as an alias in an AS. It is
> > strange to have an alias that can only be assigned by the
> language and
> > never by the user.
> >
> > Possible solutions:
> >
> > I) Status quo. We could fix it so that group is allowed to be
> > assigned as an alias in AS.
> >
> > Pros: Backward compatibility
> > Cons: a) will make the parser more complicated
> > b) see 1) above.
> >
> >
> > II) Don't give an implicit alias to the group key(s). If
> users want
> > an alias, they can assign it using AS.
> >
> > Pros: Simplicity
> > Cons: We do assign aliases to grouped bags. That is, if
> we have C =
> > GROUP B by $0 the resulting schema of C is (group, B). So
> if we don't
> > assign an alias to the group key, we now have a schema ($0,
> B). This
> > seems strange. And worse yet, if users want to alias the group
> > key(s), they'll be forced to alias all the grouped bags as well.
> >
> > III) Carry the alias (if any) that the field had before.
> So if we had
> > a script like:
> >
> > A = load 'myfile' as (x, y, z);
> > B = group A by x;
> >
> > The the schema of B would be (x, A). This is quite natural for
> > grouping of single columns. But it turns nasty when you group on
> > multiple columns. Do we then append the names to together?
> So if you
> > have
> >
> > B = group A by x, y;
> >
> > is the resulting schema (x_y, A)? Ugh.
> >
> > In this case there is also the question of what to do in
> the case of
> > cogroups, where the key may be named differently in
> different relations.
> >
> > A = load 'myfile' as (x, y, z);
> > B = load 'myotherfile' as (t, u, v);
> > C = cogroup A by x, B by t;
> >
> > Is the resulting schema (x, A, B) or (t, A, B) or are both valid?
> > This could be resolved by either saying first one always wins, or
> > allowing either.
> >
> > Pros: Very natural for the users, their fields maintain
> names through
> > the query.
> > Cons: Quickly gets burdensome in the case of multi-key groups.
> >
> > IV) Assign a non-keyword alias to the group key, like grp
> or groupkey
> > or grpkey (or some other suitable choice).
> > Pros: Least disruptive change. Users only have to go
> through their
> > scripts and find places where they use the group alias and
> change it
> > to grp (or whatever).
> > Cons: Still leaves us with a situation where we are
> assigning a name
> > to a field arbtrarily, leaving users confused as to how
> their fields
> > got named that.
> >
> > V) Remove GROUP as a keyword. It is just short for COGROUP of one
> > relation anyway.
> >
> > Pros: Smaller syntax in a language is always good.
> > Cons: Will break a lot of scripts, and confuse a lot of users who
> > only think in terms of GROUP and JOIN and never use COGROUP
> explicitly.
> >
> > One could also conceive of combinations of these. For example, we
> > always assign a name like grpkey to the group key(s), and in the
> > single key case we also carry forward the alias that the
> field already
> > had, if any.
> >
> > Thoughts? Other possibilities?
> >
> > Alan.
>
Re: Issues with group as an alias
Posted by Benjamin Reed <br...@yahoo-inc.com>.
III has open questions. Can we make a concrete proposal for III?
thanx
ben
On Friday 13 June 2008 22:24:01 Ted Dunning wrote:
> I think that I am convinced III is best.
>
> On Fri, Jun 13, 2008 at 7:26 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
> > All,
> >
> > I too will vote for III, with the caveat that we don't give names to
> > multi-field grouping keys. We need to make sure we support AS to allow
> > the user to name their grouping keys if they want.
> >
> > So far, the vote totals are:
> > I: 1
> > II: 0
> > III: 3
> > IV: 0
> > V: 0
> >
> > I'd like to make a decision and move forward by mid next week. If you
> > haven't voted and you'd like to, please do so now. If you feel
> > passionately about one of the options that is loosing, please make your
> > arguments now.
> >
> > Alan.
> >
> > Alan Gates wrote:
> >> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
> >> (or set of fields) that are grouped on are given the alias 'group'.
> >> This has a couple of issues:
> >>
> >> 1) It's confusing. 'group' is now a keyword and an alias.
> >> 2) We don't currently allow 'group' as an alias in an AS. It is
> >> strange to have an alias that can only be assigned by the language and
> >> never by the user.
> >>
> >> Possible solutions:
> >>
> >> I) Status quo. We could fix it so that group is allowed to be assigned
> >> as an alias in AS.
> >>
> >> Pros: Backward compatibility
> >> Cons: a) will make the parser more complicated
> >> b) see 1) above.
> >>
> >>
> >> II) Don't give an implicit alias to the group key(s). If users want an
> >> alias, they can assign it using AS.
> >>
> >> Pros: Simplicity
> >> Cons: We do assign aliases to grouped bags. That is, if we have C =
> >> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> >> assign an alias to the group key, we now have a schema ($0, B). This
> >> seems strange. And worse yet, if users want to alias the group key(s),
> >> they'll be forced to alias all the grouped bags as well.
> >>
> >> III) Carry the alias (if any) that the field had before. So if we had a
> >> script like:
> >>
> >> A = load 'myfile' as (x, y, z);
> >> B = group A by x;
> >>
> >> The the schema of B would be (x, A). This is quite natural for grouping
> >> of single columns. But it turns nasty when you group on multiple
> >> columns. Do we then append the names to together? So if you have
> >>
> >> B = group A by x, y;
> >>
> >> is the resulting schema (x_y, A)? Ugh.
> >>
> >> In this case there is also the question of what to do in the case of
> >> cogroups, where the key may be named differently in different relations.
> >>
> >> A = load 'myfile' as (x, y, z);
> >> B = load 'myotherfile' as (t, u, v);
> >> C = cogroup A by x, B by t;
> >>
> >> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
> >> could be resolved by either saying first one always wins, or allowing
> >> either.
> >>
> >> Pros: Very natural for the users, their fields maintain names through
> >> the query.
> >> Cons: Quickly gets burdensome in the case of multi-key groups.
> >>
> >> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
> >> grpkey (or some other suitable choice).
> >> Pros: Least disruptive change. Users only have to go through their
> >> scripts and find places where they use the group alias and change it to
> >> grp (or whatever).
> >> Cons: Still leaves us with a situation where we are assigning a name to
> >> a field arbtrarily, leaving users confused as to how their fields got
> >> named that.
> >>
> >> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> >> relation anyway.
> >>
> >> Pros: Smaller syntax in a language is always good.
> >> Cons: Will break a lot of scripts, and confuse a lot of users who only
> >> think in terms of GROUP and JOIN and never use COGROUP explicitly.
> >>
> >> One could also conceive of combinations of these. For example, we
> >> always assign a name like grpkey to the group key(s), and in the single
> >> key case we also carry forward the alias that the field already had, if
> >> any.
> >>
> >> Thoughts? Other possibilities?
> >>
> >> Alan.
Re: Issues with group as an alias
Posted by Ted Dunning <te...@gmail.com>.
I think that I am convinced III is best.
On Fri, Jun 13, 2008 at 7:26 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
> All,
>
> I too will vote for III, with the caveat that we don't give names to
> multi-field grouping keys. We need to make sure we support AS to allow the
> user to name their grouping keys if they want.
>
> So far, the vote totals are:
> I: 1
> II: 0
> III: 3
> IV: 0
> V: 0
>
> I'd like to make a decision and move forward by mid next week. If you
> haven't voted and you'd like to, please do so now. If you feel passionately
> about one of the options that is loosing, please make your arguments now.
>
> Alan.
>
> Alan Gates wrote:
>
>> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
>> (or set of fields) that are grouped on are given the alias 'group'. This
>> has a couple of issues:
>>
>> 1) It's confusing. 'group' is now a keyword and an alias.
>> 2) We don't currently allow 'group' as an alias in an AS. It is strange
>> to have an alias that can only be assigned by the language and never by the
>> user.
>>
>> Possible solutions:
>>
>> I) Status quo. We could fix it so that group is allowed to be assigned as
>> an alias in AS.
>>
>> Pros: Backward compatibility
>> Cons: a) will make the parser more complicated
>> b) see 1) above.
>>
>>
>> II) Don't give an implicit alias to the group key(s). If users want an
>> alias, they can assign it using AS.
>>
>> Pros: Simplicity
>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>> assign an alias to the group key, we now have a schema ($0, B). This seems
>> strange. And worse yet, if users want to alias the group key(s), they'll be
>> forced to alias all the grouped bags as well.
>>
>> III) Carry the alias (if any) that the field had before. So if we had a
>> script like:
>>
>> A = load 'myfile' as (x, y, z);
>> B = group A by x;
>>
>> The the schema of B would be (x, A). This is quite natural for grouping
>> of single columns. But it turns nasty when you group on multiple columns.
>> Do we then append the names to together? So if you have
>>
>> B = group A by x, y;
>>
>> is the resulting schema (x_y, A)? Ugh.
>>
>> In this case there is also the question of what to do in the case of
>> cogroups, where the key may be named differently in different relations.
>>
>> A = load 'myfile' as (x, y, z);
>> B = load 'myotherfile' as (t, u, v);
>> C = cogroup A by x, B by t;
>>
>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
>> could be resolved by either saying first one always wins, or allowing
>> either.
>>
>> Pros: Very natural for the users, their fields maintain names through the
>> query.
>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>
>> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
>> grpkey (or some other suitable choice).
>> Pros: Least disruptive change. Users only have to go through their
>> scripts and find places where they use the group alias and change it to grp
>> (or whatever).
>> Cons: Still leaves us with a situation where we are assigning a name to a
>> field arbtrarily, leaving users confused as to how their fields got named
>> that.
>>
>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>> relation anyway.
>>
>> Pros: Smaller syntax in a language is always good.
>> Cons: Will break a lot of scripts, and confuse a lot of users who only
>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>
>> One could also conceive of combinations of these. For example, we always
>> assign a name like grpkey to the group key(s), and in the single key case we
>> also carry forward the alias that the field already had, if any.
>>
>> Thoughts? Other possibilities?
>>
>> Alan.
>>
>
--
ted
Re: Issues with group as an alias
Posted by pi song <pi...@gmail.com>.
It's been over a week. Let's assume lazy vote and do it now!
On Sat, Jun 14, 2008 at 12:26 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
> All,
>
> I too will vote for III, with the caveat that we don't give names to
> multi-field grouping keys. We need to make sure we support AS to allow the
> user to name their grouping keys if they want.
>
> So far, the vote totals are:
> I: 1
> II: 0
> III: 3
> IV: 0
> V: 0
>
> I'd like to make a decision and move forward by mid next week. If you
> haven't voted and you'd like to, please do so now. If you feel passionately
> about one of the options that is loosing, please make your arguments now.
>
> Alan.
>
>
> Alan Gates wrote:
>
>> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
>> (or set of fields) that are grouped on are given the alias 'group'. This
>> has a couple of issues:
>>
>> 1) It's confusing. 'group' is now a keyword and an alias.
>> 2) We don't currently allow 'group' as an alias in an AS. It is strange
>> to have an alias that can only be assigned by the language and never by the
>> user.
>>
>> Possible solutions:
>>
>> I) Status quo. We could fix it so that group is allowed to be assigned as
>> an alias in AS.
>>
>> Pros: Backward compatibility
>> Cons: a) will make the parser more complicated
>> b) see 1) above.
>>
>>
>> II) Don't give an implicit alias to the group key(s). If users want an
>> alias, they can assign it using AS.
>>
>> Pros: Simplicity
>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>> assign an alias to the group key, we now have a schema ($0, B). This seems
>> strange. And worse yet, if users want to alias the group key(s), they'll be
>> forced to alias all the grouped bags as well.
>>
>> III) Carry the alias (if any) that the field had before. So if we had a
>> script like:
>>
>> A = load 'myfile' as (x, y, z);
>> B = group A by x;
>>
>> The the schema of B would be (x, A). This is quite natural for grouping
>> of single columns. But it turns nasty when you group on multiple columns.
>> Do we then append the names to together? So if you have
>>
>> B = group A by x, y;
>>
>> is the resulting schema (x_y, A)? Ugh.
>>
>> In this case there is also the question of what to do in the case of
>> cogroups, where the key may be named differently in different relations.
>>
>> A = load 'myfile' as (x, y, z);
>> B = load 'myotherfile' as (t, u, v);
>> C = cogroup A by x, B by t;
>>
>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
>> could be resolved by either saying first one always wins, or allowing
>> either.
>>
>> Pros: Very natural for the users, their fields maintain names through the
>> query.
>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>
>> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
>> grpkey (or some other suitable choice).
>> Pros: Least disruptive change. Users only have to go through their
>> scripts and find places where they use the group alias and change it to grp
>> (or whatever).
>> Cons: Still leaves us with a situation where we are assigning a name to a
>> field arbtrarily, leaving users confused as to how their fields got named
>> that.
>>
>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>> relation anyway.
>>
>> Pros: Smaller syntax in a language is always good.
>> Cons: Will break a lot of scripts, and confuse a lot of users who only
>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>
>> One could also conceive of combinations of these. For example, we always
>> assign a name like grpkey to the group key(s), and in the single key case we
>> also carry forward the alias that the field already had, if any.
>>
>> Thoughts? Other possibilities?
>>
>> Alan.
>>
>
Re: Issues with group as an alias
Posted by Alan Gates <ga...@yahoo-inc.com>.
All,
I too will vote for III, with the caveat that we don't give names to
multi-field grouping keys. We need to make sure we support AS to allow
the user to name their grouping keys if they want.
So far, the vote totals are:
I: 1
II: 0
III: 3
IV: 0
V: 0
I'd like to make a decision and move forward by mid next week. If you
haven't voted and you'd like to, please do so now. If you feel
passionately about one of the options that is loosing, please make your
arguments now.
Alan.
Alan Gates wrote:
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the
> field (or set of fields) that are grouped on are given the alias
> 'group'. This has a couple of issues:
>
> 1) It's confusing. 'group' is now a keyword and an alias.
> 2) We don't currently allow 'group' as an alias in an AS. It is
> strange to have an alias that can only be assigned by the language and
> never by the user.
>
> Possible solutions:
>
> I) Status quo. We could fix it so that group is allowed to be
> assigned as an alias in AS.
>
> Pros: Backward compatibility
> Cons: a) will make the parser more complicated
> b) see 1) above.
>
>
> II) Don't give an implicit alias to the group key(s). If users want
> an alias, they can assign it using AS.
>
> Pros: Simplicity
> Cons: We do assign aliases to grouped bags. That is, if we have C =
> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> assign an alias to the group key, we now have a schema ($0, B). This
> seems strange. And worse yet, if users want to alias the group
> key(s), they'll be forced to alias all the grouped bags as well.
>
> III) Carry the alias (if any) that the field had before. So if we had
> a script like:
>
> A = load 'myfile' as (x, y, z);
> B = group A by x;
>
> The the schema of B would be (x, A). This is quite natural for
> grouping of single columns. But it turns nasty when you group on
> multiple columns. Do we then append the names to together? So if you
> have
>
> B = group A by x, y;
>
> is the resulting schema (x_y, A)? Ugh.
>
> In this case there is also the question of what to do in the case of
> cogroups, where the key may be named differently in different relations.
>
> A = load 'myfile' as (x, y, z);
> B = load 'myotherfile' as (t, u, v);
> C = cogroup A by x, B by t;
>
> Is the resulting schema (x, A, B) or (t, A, B) or are both valid?
> This could be resolved by either saying first one always wins, or
> allowing either.
>
> Pros: Very natural for the users, their fields maintain names through
> the query.
> Cons: Quickly gets burdensome in the case of multi-key groups.
>
> IV) Assign a non-keyword alias to the group key, like grp or groupkey
> or grpkey (or some other suitable choice).
> Pros: Least disruptive change. Users only have to go through their
> scripts and find places where they use the group alias and change it
> to grp (or whatever).
> Cons: Still leaves us with a situation where we are assigning a name
> to a field arbtrarily, leaving users confused as to how their fields
> got named that.
>
> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> relation anyway.
>
> Pros: Smaller syntax in a language is always good.
> Cons: Will break a lot of scripts, and confuse a lot of users who
> only think in terms of GROUP and JOIN and never use COGROUP explicitly.
>
> One could also conceive of combinations of these. For example, we
> always assign a name like grpkey to the group key(s), and in the
> single key case we also carry forward the alias that the field already
> had, if any.
>
> Thoughts? Other possibilities?
>
> Alan.
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
I vote for (2), because just about every new user gets tripped up by
the implicit column names. Take for example:
A = load ...;
B = load ...;
C = cogroup A by url, B by url;
D = foreach C {
A = order A by $0;
generate flatten(A), flatten(B.$2);
};
Users think that the "order A by $0" is referring to the *outer
table* called A, not the A that is a column of C. There are two
reasons for this confusion, I think:
1. Rather than writing C.A (absolute path), the user must write A
(relative path) -- the prefix of the path is implicit.
2. The fact that C has a field called "A" is not evident from
inspecting the script -- C's schema does not appear anywhere.
Forcing the user to write:
C = cogroup A by url, B by url as (url, A, B);
(or whatever names they want to give the fields of C) would fix #2.
(We may also want to think about #1, but that's a separate issue from
the current line of discussion.)
-Chris
On Jun 17, 2008, at 7:30 AM, Benjamin Reed wrote:
> I agree with Pi. +1 for (1).
>
> ben
>
> On Tuesday 17 June 2008 03:41:13 pi song wrote:
>> If it's confusing because our model is different, people just have to
>> learn. If it's confusing because it is misleading, it has to be
>> fixed.
>>
>> As far as we can explain "why" logically, I think it should be ok.
>> I vote (1) for this.
>>
>> On Tue, Jun 17, 2008 at 8:03 AM, Chris Olston <olston@yahoo-
>> inc.com> wrote:
>>> Oh -- sorry I misunderstood.
>>>
>>> That's a valid question and now is the right time to revisit it.
>>> Does
>>> anybody see any natural naming convention *other than* naming
>>> them after
>>> the input tables (pig's current practice)? If so, let's discuss.
>>> If not,
>>> it seems the only two choices are: (1) leave it as-is, or (2) do not
>>> assign any name, and force user to use "AS" (this is what Jaql
>>> does I
>>> believe).
>>>
>>> -Chris
>>>
>>>
>>> On Jun 16, 2008, at 1:29 PM, Olga Natkovich wrote:
>>>
>>> Chris,
>>>
>>>> What I meant to ask was what do we do with the rest of the
>>>> fields in the
>>>> group tuples. Currently, we name those fields with the names of the
>>>> correspondent tables. I was asking if we want to continue that.
>>>> I know
>>>> that people find it confusing to see fields named after relations.
>>>>
>>>> Olga
>>>>
>>>> -----Original Message-----
>>>>
>>>>> From: Chris Olston [mailto:olston@yahoo-inc.com]
>>>>> Sent: Monday, June 16, 2008 12:54 PM
>>>>> To: pig-dev@incubator.apache.org
>>>>> Subject: Re: Issues with group as an alias
>>>>>
>>>>> Olga,
>>>>>
>>>>> The idea is that when there is just one field with one name,
>>>>> we use that name for the group key. In all other cases we do
>>>>> *not* supply an automatic name (the user can assign their own
>>>>> name using "as").
>>>>>
>>>>> I believe this solution: (1) is very simple and unambiguous,
>>>>> and (2) makes common cases very natural (e.g, BAR = group FOO
>>>>> by URL; foreach BAR generate URL, ...).
>>>>>
>>>>> -Chris
>>>>>
>>>>> On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
>>>>>
>>>>> What about naming the rest of the fields in the group? Do
>>>>>
>>>>> we want to
>>>>>
>>>>>> continue naming them with the names of the corresponding
>>>>>> tables? I
>>>>>> think users find that confusing as well.
>>>>>>
>>>>>> Olga
>>>>>>
>>>>>> -----Original Message-----
>>>>>>
>>>>>>> From: Alan Gates [mailto:gates@yahoo-inc.com]
>>>>>>> Sent: Monday, June 16, 2008 11:32 AM
>>>>>>> To: pig-dev@incubator.apache.org
>>>>>>> Subject: Re: Issues with group as an alias
>>>>>>>
>>>>>>> I would like to propose a slight modification:
>>>>>>>
>>>>>>> I think that we should continue to support 'group' as the
>>>>>>
>>>>>> alias name
>>>>>>
>>>>>> for some transition period (3 or maybe 6 months).
>>>>>>
>>>>>>> We can remove all references to group as an alias from the
>>>>>>> documentation and print a warning when users use it. But I
>>>>>>> don't
>>>>>>> think we should drop it immediately, as we'll break many
>>>>>>> scripts.
>>>>>>>
>>>>>>> Other than that I'm fine with the proposal.
>>>>>>>
>>>>>>> Alan.
>>>>>>>
>>>>>>> Chris Olston wrote:
>>>>>>>> No.
>>>>>>>>
>>>>>>>> The standing proposal for Option III is:
>>>>>>>>
>>>>>>>> 1. If you are (CO)Grouping on a *single* field AND in the
>>>>>>>> case of
>>>>>>>> co-group all field names are the same (e.g., cogroup A by
>>>>>>>
>>>>>>> url, B by
>>>>>>
>>>>>> url), then give the group key that name (e.g., "url").
>>>>>>
>>>>>>>> 2. Else, do *not* automatically assign any name. The user
>>>>>>>
>>>>>>> can refer to
>>>>>>>
>>>>>>>> it as $0 and/or use "AS" to give it a name manually.
>>>>>>>>
>>>>>>>> (To be clear, even in case #1, the user has the option to
>>>>>>>
>>>>>>> override the
>>>>>>>
>>>>>>>> automatically-assigned name using "AS" if s/he chooses.)
>>>>>>>>
>>>>>>>> -Chris
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>>>>>>>>
>>>>>>>> I completely agree. It does start getting confusing.
>>>>>>>>
>>>>>>>> Especially if we
>>>>>>>>
>>>>>>>> try to deal with multi field keys.
>>>>>>>>
>>>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
>>>>>>>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load
>>>>>>>>> 'somefile3'
>>>>>>>>> USING PigStorage() AS (A, B)
>>>>>>>>>
>>>>>>>>> G1 = COGROUP A by (B,C), B by (A, C);
>>>>>>>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>>>>>>>>
>>>>>>>>> What is the schema for G2?
>>>>>>>>>
>>>>>>>>> ben
>>>>>>>>>
>>>>>>>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>>>>>>>>> So what is the conclusion here ?
>>>>>>>>>>
>>>>>>>>>> group key alias == the first variables group by field ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What happens in a case like this then :
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
>>>>>>>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
>>>>>>>>>
>>>>>>>>> 'somefile3' USING
>>>>>>>>
>>>>>>>> PigStorage() AS (A, B)
>>>>>>>>
>>>>>>>>>> G1 = COGROUP A by B, B by A;
>>>>>>>>>> G2 = COGROUP A by C, C by A;
>>>>>>>>>> ...
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> A slightly contrived example for sure, but imo grammer
>>>>>>>>>
>>>>>>>>> has to be as
>>>>>>>>
>>>>>>>> clearly specified as possible.
>>>>>>>>
>>>>>>>>>> A reserved keyword as group alias implies we dont hit
>>>>>>>>>
>>>>>>>>> this problem
>>>>>>>>
>>>>>>>> (group or groupkey or grpkey)... and also the fact that we are
>>>>>>>>
>>>>>>>>>> backwardly compatible.
>>>>>>>>>>
>>>>>>>>>> [I never liked inferred schema prefix section in the
>>>>>>>>>
>>>>>>>>> schemas doc
>>>>>>
>>>>>> (which is applied selectively) - makes it extremely tough to
>>>>>>
>>>>>>>>>> generate pig scripts]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Mridul
>>>>>>>>>>
>>>>>>>>>> Alan Gates wrote:
>>>>>>>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
>>>>>>>>>>
>>>>>>>>>> used, the
>>>>>>>>
>>>>>>>> field (or set of fields) that are grouped on are given
>>>>>>>>
>>>>>>>>>> the alias
>>>>>>
>>>>>> 'group'.
>>>>>>
>>>>>>>>>>> This has a couple of issues:
>>>>>>>>>>>
>>>>>>>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>>>>>>>> 2) We don't currently allow 'group' as an alias in an
>>>>>>>>>>
>>>>>>>>>> AS. It is
>>>>>>>>
>>>>>>>> strange to have an alias that can only be assigned by
>>>>>>>>
>>>>>>>>>> the language
>>>>>>>>
>>>>>>>> and never by the user.
>>>>>>>>
>>>>>>>>>>> Possible solutions:
>>>>>>>>>>>
>>>>>>>>>>> I) Status quo. We could fix it so that group is allowed
>>>>>>>>>>> to be
>>>>>>>>>>> assigned as an alias in AS.
>>>>>>>>>>>
>>>>>>>>>>> Pros: Backward compatibility
>>>>>>>>>>> Cons: a) will make the parser more complicated
>>>>>>>>>>> b) see 1) above.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> II) Don't give an implicit alias to the group key(s).
>>>>>>>>>>
>>>>>>>>>> If users
>>>>>>
>>>>>> want an alias, they can assign it using AS.
>>>>>>
>>>>>>>>>>> Pros: Simplicity
>>>>>>>>>>> Cons: We do assign aliases to grouped bags. That is,
>>>>>>>>>>
>>>>>>>>>> if we have C
>>>>>>>>
>>>>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
>>>>>>>>
>>>>>>>>>> So if we
>>>>>>>>
>>>>>>>> don't assign an alias to the group key, we now have a
>>>>>>>>
>>>>>>>>>> schema ($0,
>>>>>>>>
>>>>>>>> B). This seems strange. And worse yet, if users want
>>>>>>>>
>>>>>>>>>> to alias the
>>>>>>>>
>>>>>>>> group key(s), they'll be forced to alias all the
>>>>>>>>
>>>>>>>>>> grouped bags as
>>>>>>
>>>>>> well.
>>>>>>
>>>>>>>>>>> III) Carry the alias (if any) that the field had before.
>>>>>>>>>>
>>>>>>>>>> So if we
>>>>>>>>
>>>>>>>> had a script like:
>>>>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>>>>> B = group A by x;
>>>>>>>>>>>
>>>>>>>>>>> The the schema of B would be (x, A). This is quite
>>>>>>>>>>
>>>>>>>>>> natural for
>>>>>>
>>>>>> grouping of single columns. But it turns nasty when you
>>>>>>
>>>>>>>>>> group on
>>>>>>>>
>>>>>>>> multiple columns. Do we then append the names to
>>>>>>>>
>>>>>>>>>> together? So if
>>>>>>>>
>>>>>>>> you have
>>>>>>>>
>>>>>>>>>>> B = group A by x, y;
>>>>>>>>>>>
>>>>>>>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>>>>>>>
>>>>>>>>>>> In this case there is also the question of what to do in
>>>>>>>>>>
>>>>>>>>>> the case
>>>>>>>>
>>>>>>>> of cogroups, where the key may be named differently in
>>>>>>>>
>>>>>>>>>> different
>>>>>>
>>>>>> relations.
>>>>>>
>>>>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by
>>>>>>>>>>
>>>>>>>>>> x, B by t;
>>>>>>>>>>
>>>>>>>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
>>>>>>>>>>
>>>>>>>>>> both valid?
>>>>>>>>
>>>>>>>> This
>>>>>>>>
>>>>>>>>>>> could be resolved by either saying first one always wins, or
>>>>>>>>>>> allowing either.
>>>>>>>>>>>
>>>>>>>>>>> Pros: Very natural for the users, their fields maintain
>>>>>>>>>>> names
>>>>>>>>>>> through the query.
>>>>>>>>>>> Cons: Quickly gets burdensome in the case of multi-key
>>>>>>>>>>> groups.
>>>>>>>>>>>
>>>>>>>>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>>>>>>>>> groupkey or grpkey (or some other suitable choice).
>>>>>>>>>>> Pros: Least disruptive change. Users only have to go
>>>>>>>>>>> through
>>>>>>>>>>> their scripts and find places where they use the group
>>>>>>>>>>
>>>>>>>>>> alias and
>>>>>>
>>>>>> change it to grp (or whatever).
>>>>>>
>>>>>>>>>>> Cons: Still leaves us with a situation where we are
>>>>>>>>>>
>>>>>>>>>> assigning a
>>>>>>
>>>>>> name to a field arbtrarily, leaving users confused as to
>>>>>>
>>>>>>>>>> how their
>>>>>>>>
>>>>>>>> fields got named that.
>>>>>>>>
>>>>>>>>>>> V) Remove GROUP as a keyword. It is just short for
>>>>>>>>>>
>>>>>>>>>> COGROUP of one
>>>>>>>>
>>>>>>>> relation anyway.
>>>>>>>>
>>>>>>>>>>> Pros: Smaller syntax in a language is always good.
>>>>>>>>>>> Cons: Will break a lot of scripts, and confuse a lot of
>>>>>>>>>>
>>>>>>>>>> users who
>>>>>>>>
>>>>>>>> only think in terms of GROUP and JOIN and never use COGROUP
>>>>>>>>
>>>>>>>>>>> explicitly.
>>>>>>>>>>>
>>>>>>>>>>> One could also conceive of combinations of these. For
>>>>>>>>>>
>>>>>>>>>> example, we
>>>>>>>>
>>>>>>>> always assign a name like grpkey to the group key(s),
>>>>>>>>
>>>>>>>>>> and in the
>>>>>>
>>>>>> single key case we also carry forward the alias that the field
>>>>>>
>>>>>>>>>>> already had, if any.
>>>>>>>>>>>
>>>>>>>>>>> Thoughts? Other possibilities?
>>>>>>>>>>>
>>>>>>>>>>> Alan.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Christopher Olston, Ph.D.
>>>>>>>> Sr. Research Scientist
>>>>>>>> Yahoo! Research
>>>>>
>>>>> --
>>>>> Christopher Olston, Ph.D.
>>>>> Sr. Research Scientist
>>>>> Yahoo! Research
>>>
>>> --
>>> Christopher Olston, Ph.D.
>>> Sr. Research Scientist
>>> Yahoo! Research
>
>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
Re: Issues with group as an alias
Posted by Benjamin Reed <br...@yahoo-inc.com>.
I agree with Pi. +1 for (1).
ben
On Tuesday 17 June 2008 03:41:13 pi song wrote:
> If it's confusing because our model is different, people just have to
> learn. If it's confusing because it is misleading, it has to be fixed.
>
> As far as we can explain "why" logically, I think it should be ok.
> I vote (1) for this.
>
> On Tue, Jun 17, 2008 at 8:03 AM, Chris Olston <ol...@yahoo-inc.com> wrote:
> > Oh -- sorry I misunderstood.
> >
> > That's a valid question and now is the right time to revisit it. Does
> > anybody see any natural naming convention *other than* naming them after
> > the input tables (pig's current practice)? If so, let's discuss. If not,
> > it seems the only two choices are: (1) leave it as-is, or (2) do not
> > assign any name, and force user to use "AS" (this is what Jaql does I
> > believe).
> >
> > -Chris
> >
> >
> > On Jun 16, 2008, at 1:29 PM, Olga Natkovich wrote:
> >
> > Chris,
> >
> >> What I meant to ask was what do we do with the rest of the fields in the
> >> group tuples. Currently, we name those fields with the names of the
> >> correspondent tables. I was asking if we want to continue that. I know
> >> that people find it confusing to see fields named after relations.
> >>
> >> Olga
> >>
> >> -----Original Message-----
> >>
> >>> From: Chris Olston [mailto:olston@yahoo-inc.com]
> >>> Sent: Monday, June 16, 2008 12:54 PM
> >>> To: pig-dev@incubator.apache.org
> >>> Subject: Re: Issues with group as an alias
> >>>
> >>> Olga,
> >>>
> >>> The idea is that when there is just one field with one name,
> >>> we use that name for the group key. In all other cases we do
> >>> *not* supply an automatic name (the user can assign their own
> >>> name using "as").
> >>>
> >>> I believe this solution: (1) is very simple and unambiguous,
> >>> and (2) makes common cases very natural (e.g, BAR = group FOO
> >>> by URL; foreach BAR generate URL, ...).
> >>>
> >>> -Chris
> >>>
> >>> On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
> >>>
> >>> What about naming the rest of the fields in the group? Do
> >>>
> >>> we want to
> >>>
> >>>> continue naming them with the names of the corresponding tables? I
> >>>> think users find that confusing as well.
> >>>>
> >>>> Olga
> >>>>
> >>>> -----Original Message-----
> >>>>
> >>>>> From: Alan Gates [mailto:gates@yahoo-inc.com]
> >>>>> Sent: Monday, June 16, 2008 11:32 AM
> >>>>> To: pig-dev@incubator.apache.org
> >>>>> Subject: Re: Issues with group as an alias
> >>>>>
> >>>>> I would like to propose a slight modification:
> >>>>>
> >>>>> I think that we should continue to support 'group' as the
> >>>>
> >>>> alias name
> >>>>
> >>>> for some transition period (3 or maybe 6 months).
> >>>>
> >>>>> We can remove all references to group as an alias from the
> >>>>> documentation and print a warning when users use it. But I don't
> >>>>> think we should drop it immediately, as we'll break many scripts.
> >>>>>
> >>>>> Other than that I'm fine with the proposal.
> >>>>>
> >>>>> Alan.
> >>>>>
> >>>>> Chris Olston wrote:
> >>>>>> No.
> >>>>>>
> >>>>>> The standing proposal for Option III is:
> >>>>>>
> >>>>>> 1. If you are (CO)Grouping on a *single* field AND in the case of
> >>>>>> co-group all field names are the same (e.g., cogroup A by
> >>>>>
> >>>>> url, B by
> >>>>
> >>>> url), then give the group key that name (e.g., "url").
> >>>>
> >>>>>> 2. Else, do *not* automatically assign any name. The user
> >>>>>
> >>>>> can refer to
> >>>>>
> >>>>>> it as $0 and/or use "AS" to give it a name manually.
> >>>>>>
> >>>>>> (To be clear, even in case #1, the user has the option to
> >>>>>
> >>>>> override the
> >>>>>
> >>>>>> automatically-assigned name using "AS" if s/he chooses.)
> >>>>>>
> >>>>>> -Chris
> >>>>>>
> >>>>>>
> >>>>>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
> >>>>>>
> >>>>>> I completely agree. It does start getting confusing.
> >>>>>>
> >>>>>> Especially if we
> >>>>>>
> >>>>>> try to deal with multi field keys.
> >>>>>>
> >>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
> >>>>>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
> >>>>>>> USING PigStorage() AS (A, B)
> >>>>>>>
> >>>>>>> G1 = COGROUP A by (B,C), B by (A, C);
> >>>>>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
> >>>>>>>
> >>>>>>> What is the schema for G2?
> >>>>>>>
> >>>>>>> ben
> >>>>>>>
> >>>>>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
> >>>>>>>> So what is the conclusion here ?
> >>>>>>>>
> >>>>>>>> group key alias == the first variables group by field ?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> What happens in a case like this then :
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
> >>>>>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
> >>>>>>>
> >>>>>>> 'somefile3' USING
> >>>>>>
> >>>>>> PigStorage() AS (A, B)
> >>>>>>
> >>>>>>>> G1 = COGROUP A by B, B by A;
> >>>>>>>> G2 = COGROUP A by C, C by A;
> >>>>>>>> ...
> >>>>>>>> --
> >>>>>>>>
> >>>>>>>> A slightly contrived example for sure, but imo grammer
> >>>>>>>
> >>>>>>> has to be as
> >>>>>>
> >>>>>> clearly specified as possible.
> >>>>>>
> >>>>>>>> A reserved keyword as group alias implies we dont hit
> >>>>>>>
> >>>>>>> this problem
> >>>>>>
> >>>>>> (group or groupkey or grpkey)... and also the fact that we are
> >>>>>>
> >>>>>>>> backwardly compatible.
> >>>>>>>>
> >>>>>>>> [I never liked inferred schema prefix section in the
> >>>>>>>
> >>>>>>> schemas doc
> >>>>
> >>>> (which is applied selectively) - makes it extremely tough to
> >>>>
> >>>>>>>> generate pig scripts]
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Mridul
> >>>>>>>>
> >>>>>>>> Alan Gates wrote:
> >>>>>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
> >>>>>>>>
> >>>>>>>> used, the
> >>>>>>
> >>>>>> field (or set of fields) that are grouped on are given
> >>>>>>
> >>>>>>>> the alias
> >>>>
> >>>> 'group'.
> >>>>
> >>>>>>>>> This has a couple of issues:
> >>>>>>>>>
> >>>>>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
> >>>>>>>>> 2) We don't currently allow 'group' as an alias in an
> >>>>>>>>
> >>>>>>>> AS. It is
> >>>>>>
> >>>>>> strange to have an alias that can only be assigned by
> >>>>>>
> >>>>>>>> the language
> >>>>>>
> >>>>>> and never by the user.
> >>>>>>
> >>>>>>>>> Possible solutions:
> >>>>>>>>>
> >>>>>>>>> I) Status quo. We could fix it so that group is allowed to be
> >>>>>>>>> assigned as an alias in AS.
> >>>>>>>>>
> >>>>>>>>> Pros: Backward compatibility
> >>>>>>>>> Cons: a) will make the parser more complicated
> >>>>>>>>> b) see 1) above.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> II) Don't give an implicit alias to the group key(s).
> >>>>>>>>
> >>>>>>>> If users
> >>>>
> >>>> want an alias, they can assign it using AS.
> >>>>
> >>>>>>>>> Pros: Simplicity
> >>>>>>>>> Cons: We do assign aliases to grouped bags. That is,
> >>>>>>>>
> >>>>>>>> if we have C
> >>>>>>
> >>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
> >>>>>>
> >>>>>>>> So if we
> >>>>>>
> >>>>>> don't assign an alias to the group key, we now have a
> >>>>>>
> >>>>>>>> schema ($0,
> >>>>>>
> >>>>>> B). This seems strange. And worse yet, if users want
> >>>>>>
> >>>>>>>> to alias the
> >>>>>>
> >>>>>> group key(s), they'll be forced to alias all the
> >>>>>>
> >>>>>>>> grouped bags as
> >>>>
> >>>> well.
> >>>>
> >>>>>>>>> III) Carry the alias (if any) that the field had before.
> >>>>>>>>
> >>>>>>>> So if we
> >>>>>>
> >>>>>> had a script like:
> >>>>>>>>> A = load 'myfile' as (x, y, z);
> >>>>>>>>> B = group A by x;
> >>>>>>>>>
> >>>>>>>>> The the schema of B would be (x, A). This is quite
> >>>>>>>>
> >>>>>>>> natural for
> >>>>
> >>>> grouping of single columns. But it turns nasty when you
> >>>>
> >>>>>>>> group on
> >>>>>>
> >>>>>> multiple columns. Do we then append the names to
> >>>>>>
> >>>>>>>> together? So if
> >>>>>>
> >>>>>> you have
> >>>>>>
> >>>>>>>>> B = group A by x, y;
> >>>>>>>>>
> >>>>>>>>> is the resulting schema (x_y, A)? Ugh.
> >>>>>>>>>
> >>>>>>>>> In this case there is also the question of what to do in
> >>>>>>>>
> >>>>>>>> the case
> >>>>>>
> >>>>>> of cogroups, where the key may be named differently in
> >>>>>>
> >>>>>>>> different
> >>>>
> >>>> relations.
> >>>>
> >>>>>>>>> A = load 'myfile' as (x, y, z);
> >>>>>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by
> >>>>>>>>
> >>>>>>>> x, B by t;
> >>>>>>>>
> >>>>>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
> >>>>>>>>
> >>>>>>>> both valid?
> >>>>>>
> >>>>>> This
> >>>>>>
> >>>>>>>>> could be resolved by either saying first one always wins, or
> >>>>>>>>> allowing either.
> >>>>>>>>>
> >>>>>>>>> Pros: Very natural for the users, their fields maintain names
> >>>>>>>>> through the query.
> >>>>>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
> >>>>>>>>>
> >>>>>>>>> IV) Assign a non-keyword alias to the group key, like grp or
> >>>>>>>>> groupkey or grpkey (or some other suitable choice).
> >>>>>>>>> Pros: Least disruptive change. Users only have to go through
> >>>>>>>>> their scripts and find places where they use the group
> >>>>>>>>
> >>>>>>>> alias and
> >>>>
> >>>> change it to grp (or whatever).
> >>>>
> >>>>>>>>> Cons: Still leaves us with a situation where we are
> >>>>>>>>
> >>>>>>>> assigning a
> >>>>
> >>>> name to a field arbtrarily, leaving users confused as to
> >>>>
> >>>>>>>> how their
> >>>>>>
> >>>>>> fields got named that.
> >>>>>>
> >>>>>>>>> V) Remove GROUP as a keyword. It is just short for
> >>>>>>>>
> >>>>>>>> COGROUP of one
> >>>>>>
> >>>>>> relation anyway.
> >>>>>>
> >>>>>>>>> Pros: Smaller syntax in a language is always good.
> >>>>>>>>> Cons: Will break a lot of scripts, and confuse a lot of
> >>>>>>>>
> >>>>>>>> users who
> >>>>>>
> >>>>>> only think in terms of GROUP and JOIN and never use COGROUP
> >>>>>>
> >>>>>>>>> explicitly.
> >>>>>>>>>
> >>>>>>>>> One could also conceive of combinations of these. For
> >>>>>>>>
> >>>>>>>> example, we
> >>>>>>
> >>>>>> always assign a name like grpkey to the group key(s),
> >>>>>>
> >>>>>>>> and in the
> >>>>
> >>>> single key case we also carry forward the alias that the field
> >>>>
> >>>>>>>>> already had, if any.
> >>>>>>>>>
> >>>>>>>>> Thoughts? Other possibilities?
> >>>>>>>>>
> >>>>>>>>> Alan.
> >>>>>>
> >>>>>> --
> >>>>>> Christopher Olston, Ph.D.
> >>>>>> Sr. Research Scientist
> >>>>>> Yahoo! Research
> >>>
> >>> --
> >>> Christopher Olston, Ph.D.
> >>> Sr. Research Scientist
> >>> Yahoo! Research
> >
> > --
> > Christopher Olston, Ph.D.
> > Sr. Research Scientist
> > Yahoo! Research
Re: Issues with group as an alias
Posted by pi song <pi...@gmail.com>.
If it's confusing because our model is different, people just have to learn.
If it's confusing because it is misleading, it has to be fixed.
As far as we can explain "why" logically, I think it should be ok.
I vote (1) for this.
On Tue, Jun 17, 2008 at 8:03 AM, Chris Olston <ol...@yahoo-inc.com> wrote:
> Oh -- sorry I misunderstood.
>
> That's a valid question and now is the right time to revisit it. Does
> anybody see any natural naming convention *other than* naming them after the
> input tables (pig's current practice)? If so, let's discuss. If not, it
> seems the only two choices are: (1) leave it as-is, or (2) do not assign any
> name, and force user to use "AS" (this is what Jaql does I believe).
>
> -Chris
>
>
> On Jun 16, 2008, at 1:29 PM, Olga Natkovich wrote:
>
> Chris,
>>
>> What I meant to ask was what do we do with the rest of the fields in the
>> group tuples. Currently, we name those fields with the names of the
>> correspondent tables. I was asking if we want to continue that. I know
>> that people find it confusing to see fields named after relations.
>>
>> Olga
>>
>> -----Original Message-----
>>> From: Chris Olston [mailto:olston@yahoo-inc.com]
>>> Sent: Monday, June 16, 2008 12:54 PM
>>> To: pig-dev@incubator.apache.org
>>> Subject: Re: Issues with group as an alias
>>>
>>> Olga,
>>>
>>> The idea is that when there is just one field with one name,
>>> we use that name for the group key. In all other cases we do
>>> *not* supply an automatic name (the user can assign their own
>>> name using "as").
>>>
>>> I believe this solution: (1) is very simple and unambiguous,
>>> and (2) makes common cases very natural (e.g, BAR = group FOO
>>> by URL; foreach BAR generate URL, ...).
>>>
>>> -Chris
>>>
>>> On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
>>>
>>> What about naming the rest of the fields in the group? Do
>>>>
>>> we want to
>>>
>>>> continue naming them with the names of the corresponding tables? I
>>>> think users find that confusing as well.
>>>>
>>>> Olga
>>>>
>>>> -----Original Message-----
>>>>> From: Alan Gates [mailto:gates@yahoo-inc.com]
>>>>> Sent: Monday, June 16, 2008 11:32 AM
>>>>> To: pig-dev@incubator.apache.org
>>>>> Subject: Re: Issues with group as an alias
>>>>>
>>>>> I would like to propose a slight modification:
>>>>>
>>>>> I think that we should continue to support 'group' as the
>>>>>
>>>> alias name
>>>
>>>> for some transition period (3 or maybe 6 months).
>>>>> We can remove all references to group as an alias from the
>>>>> documentation and print a warning when users use it. But I don't
>>>>> think we should drop it immediately, as we'll break many scripts.
>>>>>
>>>>> Other than that I'm fine with the proposal.
>>>>>
>>>>> Alan.
>>>>>
>>>>> Chris Olston wrote:
>>>>>
>>>>>> No.
>>>>>>
>>>>>> The standing proposal for Option III is:
>>>>>>
>>>>>> 1. If you are (CO)Grouping on a *single* field AND in the case of
>>>>>> co-group all field names are the same (e.g., cogroup A by
>>>>>>
>>>>> url, B by
>>>
>>>> url), then give the group key that name (e.g., "url").
>>>>>> 2. Else, do *not* automatically assign any name. The user
>>>>>>
>>>>> can refer to
>>>>>
>>>>>> it as $0 and/or use "AS" to give it a name manually.
>>>>>>
>>>>>> (To be clear, even in case #1, the user has the option to
>>>>>>
>>>>> override the
>>>>>
>>>>>> automatically-assigned name using "AS" if s/he chooses.)
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>>
>>>>>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>>>>>>
>>>>>> I completely agree. It does start getting confusing.
>>>>>>>
>>>>>> Especially if we
>>>>>
>>>>>> try to deal with multi field keys.
>>>>>>>
>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
>>>>>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
>>>>>>> USING PigStorage() AS (A, B)
>>>>>>>
>>>>>>> G1 = COGROUP A by (B,C), B by (A, C);
>>>>>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>>>>>>
>>>>>>> What is the schema for G2?
>>>>>>>
>>>>>>> ben
>>>>>>>
>>>>>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>>>>>>
>>>>>>>> So what is the conclusion here ?
>>>>>>>>
>>>>>>>> group key alias == the first variables group by field ?
>>>>>>>>
>>>>>>>>
>>>>>>>> What happens in a case like this then :
>>>>>>>>
>>>>>>>> --
>>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
>>>>>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
>>>>>>>>
>>>>>>> 'somefile3' USING
>>>>>
>>>>>> PigStorage() AS (A, B)
>>>>>>>>
>>>>>>>> G1 = COGROUP A by B, B by A;
>>>>>>>> G2 = COGROUP A by C, C by A;
>>>>>>>> ...
>>>>>>>> --
>>>>>>>>
>>>>>>>> A slightly contrived example for sure, but imo grammer
>>>>>>>>
>>>>>>> has to be as
>>>>>
>>>>>> clearly specified as possible.
>>>>>>>>
>>>>>>>> A reserved keyword as group alias implies we dont hit
>>>>>>>>
>>>>>>> this problem
>>>>>
>>>>>> (group or groupkey or grpkey)... and also the fact that we are
>>>>>>>> backwardly compatible.
>>>>>>>>
>>>>>>>> [I never liked inferred schema prefix section in the
>>>>>>>>
>>>>>>> schemas doc
>>>
>>>> (which is applied selectively) - makes it extremely tough to
>>>>>>>> generate pig scripts]
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Mridul
>>>>>>>>
>>>>>>>> Alan Gates wrote:
>>>>>>>>
>>>>>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
>>>>>>>>>
>>>>>>>> used, the
>>>>>
>>>>>> field (or set of fields) that are grouped on are given
>>>>>>>>>
>>>>>>>> the alias
>>>
>>>> 'group'.
>>>>>>>>> This has a couple of issues:
>>>>>>>>>
>>>>>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>>>>>> 2) We don't currently allow 'group' as an alias in an
>>>>>>>>>
>>>>>>>> AS. It is
>>>>>
>>>>>> strange to have an alias that can only be assigned by
>>>>>>>>>
>>>>>>>> the language
>>>>>
>>>>>> and never by the user.
>>>>>>>>>
>>>>>>>>> Possible solutions:
>>>>>>>>>
>>>>>>>>> I) Status quo. We could fix it so that group is allowed to be
>>>>>>>>> assigned as an alias in AS.
>>>>>>>>>
>>>>>>>>> Pros: Backward compatibility
>>>>>>>>> Cons: a) will make the parser more complicated
>>>>>>>>> b) see 1) above.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> II) Don't give an implicit alias to the group key(s).
>>>>>>>>>
>>>>>>>> If users
>>>
>>>> want an alias, they can assign it using AS.
>>>>>>>>>
>>>>>>>>> Pros: Simplicity
>>>>>>>>> Cons: We do assign aliases to grouped bags. That is,
>>>>>>>>>
>>>>>>>> if we have C
>>>>>
>>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
>>>>>>>>>
>>>>>>>> So if we
>>>>>
>>>>>> don't assign an alias to the group key, we now have a
>>>>>>>>>
>>>>>>>> schema ($0,
>>>>>
>>>>>> B). This seems strange. And worse yet, if users want
>>>>>>>>>
>>>>>>>> to alias the
>>>>>
>>>>>> group key(s), they'll be forced to alias all the
>>>>>>>>>
>>>>>>>> grouped bags as
>>>
>>>> well.
>>>>>>>>>
>>>>>>>>> III) Carry the alias (if any) that the field had before.
>>>>>>>>>
>>>>>>>> So if we
>>>>>
>>>>>> had a script like:
>>>>>>>>>
>>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>>> B = group A by x;
>>>>>>>>>
>>>>>>>>> The the schema of B would be (x, A). This is quite
>>>>>>>>>
>>>>>>>> natural for
>>>
>>>> grouping of single columns. But it turns nasty when you
>>>>>>>>>
>>>>>>>> group on
>>>>>
>>>>>> multiple columns. Do we then append the names to
>>>>>>>>>
>>>>>>>> together? So if
>>>>>
>>>>>> you have
>>>>>>>>>
>>>>>>>>> B = group A by x, y;
>>>>>>>>>
>>>>>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>>>>>
>>>>>>>>> In this case there is also the question of what to do in
>>>>>>>>>
>>>>>>>> the case
>>>>>
>>>>>> of cogroups, where the key may be named differently in
>>>>>>>>>
>>>>>>>> different
>>>
>>>> relations.
>>>>>>>>>
>>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by
>>>>>>>>>
>>>>>>>> x, B by t;
>>>
>>>>
>>>>>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
>>>>>>>>>
>>>>>>>> both valid?
>>>>>
>>>>>> This
>>>>>>>>> could be resolved by either saying first one always wins, or
>>>>>>>>> allowing either.
>>>>>>>>>
>>>>>>>>> Pros: Very natural for the users, their fields maintain names
>>>>>>>>> through the query.
>>>>>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>>>>>>
>>>>>>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>>>>>>> groupkey or grpkey (or some other suitable choice).
>>>>>>>>> Pros: Least disruptive change. Users only have to go through
>>>>>>>>> their scripts and find places where they use the group
>>>>>>>>>
>>>>>>>> alias and
>>>
>>>> change it to grp (or whatever).
>>>>>>>>> Cons: Still leaves us with a situation where we are
>>>>>>>>>
>>>>>>>> assigning a
>>>
>>>> name to a field arbtrarily, leaving users confused as to
>>>>>>>>>
>>>>>>>> how their
>>>>>
>>>>>> fields got named that.
>>>>>>>>>
>>>>>>>>> V) Remove GROUP as a keyword. It is just short for
>>>>>>>>>
>>>>>>>> COGROUP of one
>>>>>
>>>>>> relation anyway.
>>>>>>>>>
>>>>>>>>> Pros: Smaller syntax in a language is always good.
>>>>>>>>> Cons: Will break a lot of scripts, and confuse a lot of
>>>>>>>>>
>>>>>>>> users who
>>>>>
>>>>>> only think in terms of GROUP and JOIN and never use COGROUP
>>>>>>>>> explicitly.
>>>>>>>>>
>>>>>>>>> One could also conceive of combinations of these. For
>>>>>>>>>
>>>>>>>> example, we
>>>>>
>>>>>> always assign a name like grpkey to the group key(s),
>>>>>>>>>
>>>>>>>> and in the
>>>
>>>> single key case we also carry forward the alias that the field
>>>>>>>>> already had, if any.
>>>>>>>>>
>>>>>>>>> Thoughts? Other possibilities?
>>>>>>>>>
>>>>>>>>> Alan.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Christopher Olston, Ph.D.
>>>>>> Sr. Research Scientist
>>>>>> Yahoo! Research
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>> --
>>> Christopher Olston, Ph.D.
>>> Sr. Research Scientist
>>> Yahoo! Research
>>>
>>>
>>>
>>>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
Oh -- sorry I misunderstood.
That's a valid question and now is the right time to revisit it. Does
anybody see any natural naming convention *other than* naming them
after the input tables (pig's current practice)? If so, let's
discuss. If not, it seems the only two choices are: (1) leave it as-
is, or (2) do not assign any name, and force user to use "AS" (this
is what Jaql does I believe).
-Chris
On Jun 16, 2008, at 1:29 PM, Olga Natkovich wrote:
> Chris,
>
> What I meant to ask was what do we do with the rest of the fields
> in the
> group tuples. Currently, we name those fields with the names of the
> correspondent tables. I was asking if we want to continue that. I know
> that people find it confusing to see fields named after relations.
>
> Olga
>
>> -----Original Message-----
>> From: Chris Olston [mailto:olston@yahoo-inc.com]
>> Sent: Monday, June 16, 2008 12:54 PM
>> To: pig-dev@incubator.apache.org
>> Subject: Re: Issues with group as an alias
>>
>> Olga,
>>
>> The idea is that when there is just one field with one name,
>> we use that name for the group key. In all other cases we do
>> *not* supply an automatic name (the user can assign their own
>> name using "as").
>>
>> I believe this solution: (1) is very simple and unambiguous,
>> and (2) makes common cases very natural (e.g, BAR = group FOO
>> by URL; foreach BAR generate URL, ...).
>>
>> -Chris
>>
>> On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
>>
>>> What about naming the rest of the fields in the group? Do
>> we want to
>>> continue naming them with the names of the corresponding tables? I
>>> think users find that confusing as well.
>>>
>>> Olga
>>>
>>>> -----Original Message-----
>>>> From: Alan Gates [mailto:gates@yahoo-inc.com]
>>>> Sent: Monday, June 16, 2008 11:32 AM
>>>> To: pig-dev@incubator.apache.org
>>>> Subject: Re: Issues with group as an alias
>>>>
>>>> I would like to propose a slight modification:
>>>>
>>>> I think that we should continue to support 'group' as the
>> alias name
>>>> for some transition period (3 or maybe 6 months).
>>>> We can remove all references to group as an alias from the
>>>> documentation and print a warning when users use it. But I don't
>>>> think we should drop it immediately, as we'll break many scripts.
>>>>
>>>> Other than that I'm fine with the proposal.
>>>>
>>>> Alan.
>>>>
>>>> Chris Olston wrote:
>>>>> No.
>>>>>
>>>>> The standing proposal for Option III is:
>>>>>
>>>>> 1. If you are (CO)Grouping on a *single* field AND in the case of
>>>>> co-group all field names are the same (e.g., cogroup A by
>> url, B by
>>>>> url), then give the group key that name (e.g., "url").
>>>>> 2. Else, do *not* automatically assign any name. The user
>>>> can refer to
>>>>> it as $0 and/or use "AS" to give it a name manually.
>>>>>
>>>>> (To be clear, even in case #1, the user has the option to
>>>> override the
>>>>> automatically-assigned name using "AS" if s/he chooses.)
>>>>>
>>>>> -Chris
>>>>>
>>>>>
>>>>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>>>>>
>>>>>> I completely agree. It does start getting confusing.
>>>> Especially if we
>>>>>> try to deal with multi field keys.
>>>>>>
>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
>>>>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
>>>>>> USING PigStorage() AS (A, B)
>>>>>>
>>>>>> G1 = COGROUP A by (B,C), B by (A, C);
>>>>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>>>>>
>>>>>> What is the schema for G2?
>>>>>>
>>>>>> ben
>>>>>>
>>>>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>>>>>> So what is the conclusion here ?
>>>>>>>
>>>>>>> group key alias == the first variables group by field ?
>>>>>>>
>>>>>>>
>>>>>>> What happens in a case like this then :
>>>>>>>
>>>>>>> --
>>>>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
>>>>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
>>>> 'somefile3' USING
>>>>>>> PigStorage() AS (A, B)
>>>>>>>
>>>>>>> G1 = COGROUP A by B, B by A;
>>>>>>> G2 = COGROUP A by C, C by A;
>>>>>>> ...
>>>>>>> --
>>>>>>>
>>>>>>> A slightly contrived example for sure, but imo grammer
>>>> has to be as
>>>>>>> clearly specified as possible.
>>>>>>>
>>>>>>> A reserved keyword as group alias implies we dont hit
>>>> this problem
>>>>>>> (group or groupkey or grpkey)... and also the fact that we are
>>>>>>> backwardly compatible.
>>>>>>>
>>>>>>> [I never liked inferred schema prefix section in the
>> schemas doc
>>>>>>> (which is applied selectively) - makes it extremely tough to
>>>>>>> generate pig scripts]
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mridul
>>>>>>>
>>>>>>> Alan Gates wrote:
>>>>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
>>>> used, the
>>>>>>>> field (or set of fields) that are grouped on are given
>> the alias
>>>>>>>> 'group'.
>>>>>>>> This has a couple of issues:
>>>>>>>>
>>>>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>>>>> 2) We don't currently allow 'group' as an alias in an
>>>> AS. It is
>>>>>>>> strange to have an alias that can only be assigned by
>>>> the language
>>>>>>>> and never by the user.
>>>>>>>>
>>>>>>>> Possible solutions:
>>>>>>>>
>>>>>>>> I) Status quo. We could fix it so that group is allowed to be
>>>>>>>> assigned as an alias in AS.
>>>>>>>>
>>>>>>>> Pros: Backward compatibility
>>>>>>>> Cons: a) will make the parser more complicated
>>>>>>>> b) see 1) above.
>>>>>>>>
>>>>>>>>
>>>>>>>> II) Don't give an implicit alias to the group key(s).
>> If users
>>>>>>>> want an alias, they can assign it using AS.
>>>>>>>>
>>>>>>>> Pros: Simplicity
>>>>>>>> Cons: We do assign aliases to grouped bags. That is,
>>>> if we have C
>>>>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
>>>> So if we
>>>>>>>> don't assign an alias to the group key, we now have a
>>>> schema ($0,
>>>>>>>> B). This seems strange. And worse yet, if users want
>>>> to alias the
>>>>>>>> group key(s), they'll be forced to alias all the
>> grouped bags as
>>>>>>>> well.
>>>>>>>>
>>>>>>>> III) Carry the alias (if any) that the field had before.
>>>> So if we
>>>>>>>> had a script like:
>>>>>>>>
>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>> B = group A by x;
>>>>>>>>
>>>>>>>> The the schema of B would be (x, A). This is quite
>> natural for
>>>>>>>> grouping of single columns. But it turns nasty when you
>>>> group on
>>>>>>>> multiple columns. Do we then append the names to
>>>> together? So if
>>>>>>>> you have
>>>>>>>>
>>>>>>>> B = group A by x, y;
>>>>>>>>
>>>>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>>>>
>>>>>>>> In this case there is also the question of what to do in
>>>> the case
>>>>>>>> of cogroups, where the key may be named differently in
>> different
>>>>>>>> relations.
>>>>>>>>
>>>>>>>> A = load 'myfile' as (x, y, z);
>>>>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by
>> x, B by t;
>>>>>>>>
>>>>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
>>>> both valid?
>>>>>>>> This
>>>>>>>> could be resolved by either saying first one always wins, or
>>>>>>>> allowing either.
>>>>>>>>
>>>>>>>> Pros: Very natural for the users, their fields maintain names
>>>>>>>> through the query.
>>>>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>>>>>
>>>>>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>>>>>> groupkey or grpkey (or some other suitable choice).
>>>>>>>> Pros: Least disruptive change. Users only have to go through
>>>>>>>> their scripts and find places where they use the group
>> alias and
>>>>>>>> change it to grp (or whatever).
>>>>>>>> Cons: Still leaves us with a situation where we are
>> assigning a
>>>>>>>> name to a field arbtrarily, leaving users confused as to
>>>> how their
>>>>>>>> fields got named that.
>>>>>>>>
>>>>>>>> V) Remove GROUP as a keyword. It is just short for
>>>> COGROUP of one
>>>>>>>> relation anyway.
>>>>>>>>
>>>>>>>> Pros: Smaller syntax in a language is always good.
>>>>>>>> Cons: Will break a lot of scripts, and confuse a lot of
>>>> users who
>>>>>>>> only think in terms of GROUP and JOIN and never use COGROUP
>>>>>>>> explicitly.
>>>>>>>>
>>>>>>>> One could also conceive of combinations of these. For
>>>> example, we
>>>>>>>> always assign a name like grpkey to the group key(s),
>> and in the
>>>>>>>> single key case we also carry forward the alias that the field
>>>>>>>> already had, if any.
>>>>>>>>
>>>>>>>> Thoughts? Other possibilities?
>>>>>>>>
>>>>>>>> Alan.
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Christopher Olston, Ph.D.
>>>>> Sr. Research Scientist
>>>>> Yahoo! Research
>>>>>
>>>>>
>>>>>
>>>>
>>
>> --
>> Christopher Olston, Ph.D.
>> Sr. Research Scientist
>> Yahoo! Research
>>
>>
>>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
RE: Issues with group as an alias
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
Chris,
What I meant to ask was what do we do with the rest of the fields in the
group tuples. Currently, we name those fields with the names of the
correspondent tables. I was asking if we want to continue that. I know
that people find it confusing to see fields named after relations.
Olga
> -----Original Message-----
> From: Chris Olston [mailto:olston@yahoo-inc.com]
> Sent: Monday, June 16, 2008 12:54 PM
> To: pig-dev@incubator.apache.org
> Subject: Re: Issues with group as an alias
>
> Olga,
>
> The idea is that when there is just one field with one name,
> we use that name for the group key. In all other cases we do
> *not* supply an automatic name (the user can assign their own
> name using "as").
>
> I believe this solution: (1) is very simple and unambiguous,
> and (2) makes common cases very natural (e.g, BAR = group FOO
> by URL; foreach BAR generate URL, ...).
>
> -Chris
>
> On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
>
> > What about naming the rest of the fields in the group? Do
> we want to
> > continue naming them with the names of the corresponding tables? I
> > think users find that confusing as well.
> >
> > Olga
> >
> >> -----Original Message-----
> >> From: Alan Gates [mailto:gates@yahoo-inc.com]
> >> Sent: Monday, June 16, 2008 11:32 AM
> >> To: pig-dev@incubator.apache.org
> >> Subject: Re: Issues with group as an alias
> >>
> >> I would like to propose a slight modification:
> >>
> >> I think that we should continue to support 'group' as the
> alias name
> >> for some transition period (3 or maybe 6 months).
> >> We can remove all references to group as an alias from the
> >> documentation and print a warning when users use it. But I don't
> >> think we should drop it immediately, as we'll break many scripts.
> >>
> >> Other than that I'm fine with the proposal.
> >>
> >> Alan.
> >>
> >> Chris Olston wrote:
> >>> No.
> >>>
> >>> The standing proposal for Option III is:
> >>>
> >>> 1. If you are (CO)Grouping on a *single* field AND in the case of
> >>> co-group all field names are the same (e.g., cogroup A by
> url, B by
> >>> url), then give the group key that name (e.g., "url").
> >>> 2. Else, do *not* automatically assign any name. The user
> >> can refer to
> >>> it as $0 and/or use "AS" to give it a name manually.
> >>>
> >>> (To be clear, even in case #1, the user has the option to
> >> override the
> >>> automatically-assigned name using "AS" if s/he chooses.)
> >>>
> >>> -Chris
> >>>
> >>>
> >>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
> >>>
> >>>> I completely agree. It does start getting confusing.
> >> Especially if we
> >>>> try to deal with multi field keys.
> >>>>
> >>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
> >>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
> >>>> USING PigStorage() AS (A, B)
> >>>>
> >>>> G1 = COGROUP A by (B,C), B by (A, C);
> >>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
> >>>>
> >>>> What is the schema for G2?
> >>>>
> >>>> ben
> >>>>
> >>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
> >>>>> So what is the conclusion here ?
> >>>>>
> >>>>> group key alias == the first variables group by field ?
> >>>>>
> >>>>>
> >>>>> What happens in a case like this then :
> >>>>>
> >>>>> --
> >>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
> >>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
> >> 'somefile3' USING
> >>>>> PigStorage() AS (A, B)
> >>>>>
> >>>>> G1 = COGROUP A by B, B by A;
> >>>>> G2 = COGROUP A by C, C by A;
> >>>>> ...
> >>>>> --
> >>>>>
> >>>>> A slightly contrived example for sure, but imo grammer
> >> has to be as
> >>>>> clearly specified as possible.
> >>>>>
> >>>>> A reserved keyword as group alias implies we dont hit
> >> this problem
> >>>>> (group or groupkey or grpkey)... and also the fact that we are
> >>>>> backwardly compatible.
> >>>>>
> >>>>> [I never liked inferred schema prefix section in the
> schemas doc
> >>>>> (which is applied selectively) - makes it extremely tough to
> >>>>> generate pig scripts]
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Mridul
> >>>>>
> >>>>> Alan Gates wrote:
> >>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
> >> used, the
> >>>>>> field (or set of fields) that are grouped on are given
> the alias
> >>>>>> 'group'.
> >>>>>> This has a couple of issues:
> >>>>>>
> >>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
> >>>>>> 2) We don't currently allow 'group' as an alias in an
> >> AS. It is
> >>>>>> strange to have an alias that can only be assigned by
> >> the language
> >>>>>> and never by the user.
> >>>>>>
> >>>>>> Possible solutions:
> >>>>>>
> >>>>>> I) Status quo. We could fix it so that group is allowed to be
> >>>>>> assigned as an alias in AS.
> >>>>>>
> >>>>>> Pros: Backward compatibility
> >>>>>> Cons: a) will make the parser more complicated
> >>>>>> b) see 1) above.
> >>>>>>
> >>>>>>
> >>>>>> II) Don't give an implicit alias to the group key(s).
> If users
> >>>>>> want an alias, they can assign it using AS.
> >>>>>>
> >>>>>> Pros: Simplicity
> >>>>>> Cons: We do assign aliases to grouped bags. That is,
> >> if we have C
> >>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
> >> So if we
> >>>>>> don't assign an alias to the group key, we now have a
> >> schema ($0,
> >>>>>> B). This seems strange. And worse yet, if users want
> >> to alias the
> >>>>>> group key(s), they'll be forced to alias all the
> grouped bags as
> >>>>>> well.
> >>>>>>
> >>>>>> III) Carry the alias (if any) that the field had before.
> >> So if we
> >>>>>> had a script like:
> >>>>>>
> >>>>>> A = load 'myfile' as (x, y, z);
> >>>>>> B = group A by x;
> >>>>>>
> >>>>>> The the schema of B would be (x, A). This is quite
> natural for
> >>>>>> grouping of single columns. But it turns nasty when you
> >> group on
> >>>>>> multiple columns. Do we then append the names to
> >> together? So if
> >>>>>> you have
> >>>>>>
> >>>>>> B = group A by x, y;
> >>>>>>
> >>>>>> is the resulting schema (x_y, A)? Ugh.
> >>>>>>
> >>>>>> In this case there is also the question of what to do in
> >> the case
> >>>>>> of cogroups, where the key may be named differently in
> different
> >>>>>> relations.
> >>>>>>
> >>>>>> A = load 'myfile' as (x, y, z);
> >>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by
> x, B by t;
> >>>>>>
> >>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
> >> both valid?
> >>>>>> This
> >>>>>> could be resolved by either saying first one always wins, or
> >>>>>> allowing either.
> >>>>>>
> >>>>>> Pros: Very natural for the users, their fields maintain names
> >>>>>> through the query.
> >>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
> >>>>>>
> >>>>>> IV) Assign a non-keyword alias to the group key, like grp or
> >>>>>> groupkey or grpkey (or some other suitable choice).
> >>>>>> Pros: Least disruptive change. Users only have to go through
> >>>>>> their scripts and find places where they use the group
> alias and
> >>>>>> change it to grp (or whatever).
> >>>>>> Cons: Still leaves us with a situation where we are
> assigning a
> >>>>>> name to a field arbtrarily, leaving users confused as to
> >> how their
> >>>>>> fields got named that.
> >>>>>>
> >>>>>> V) Remove GROUP as a keyword. It is just short for
> >> COGROUP of one
> >>>>>> relation anyway.
> >>>>>>
> >>>>>> Pros: Smaller syntax in a language is always good.
> >>>>>> Cons: Will break a lot of scripts, and confuse a lot of
> >> users who
> >>>>>> only think in terms of GROUP and JOIN and never use COGROUP
> >>>>>> explicitly.
> >>>>>>
> >>>>>> One could also conceive of combinations of these. For
> >> example, we
> >>>>>> always assign a name like grpkey to the group key(s),
> and in the
> >>>>>> single key case we also carry forward the alias that the field
> >>>>>> already had, if any.
> >>>>>>
> >>>>>> Thoughts? Other possibilities?
> >>>>>>
> >>>>>> Alan.
> >>>>
> >>>>
> >>>
> >>> --
> >>> Christopher Olston, Ph.D.
> >>> Sr. Research Scientist
> >>> Yahoo! Research
> >>>
> >>>
> >>>
> >>
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
Olga,
The idea is that when there is just one field with one name, we use
that name for the group key. In all other cases we do *not* supply an
automatic name (the user can assign their own name using "as").
I believe this solution: (1) is very simple and unambiguous, and (2)
makes common cases very natural (e.g, BAR = group FOO by URL; foreach
BAR generate URL, ...).
-Chris
On Jun 16, 2008, at 12:48 PM, Olga Natkovich wrote:
> What about naming the rest of the fields in the group? Do we want to
> continue naming them with the names of the corresponding tables? I
> think
> users find that confusing as well.
>
> Olga
>
>> -----Original Message-----
>> From: Alan Gates [mailto:gates@yahoo-inc.com]
>> Sent: Monday, June 16, 2008 11:32 AM
>> To: pig-dev@incubator.apache.org
>> Subject: Re: Issues with group as an alias
>>
>> I would like to propose a slight modification:
>>
>> I think that we should continue to support 'group' as the
>> alias name for some transition period (3 or maybe 6 months).
>> We can remove all references to group as an alias from the
>> documentation and print a warning when users use it. But I
>> don't think we should drop it immediately, as we'll break
>> many scripts.
>>
>> Other than that I'm fine with the proposal.
>>
>> Alan.
>>
>> Chris Olston wrote:
>>> No.
>>>
>>> The standing proposal for Option III is:
>>>
>>> 1. If you are (CO)Grouping on a *single* field AND in the case of
>>> co-group all field names are the same (e.g., cogroup A by url, B by
>>> url), then give the group key that name (e.g., "url").
>>> 2. Else, do *not* automatically assign any name. The user
>> can refer to
>>> it as $0 and/or use "AS" to give it a name manually.
>>>
>>> (To be clear, even in case #1, the user has the option to
>> override the
>>> automatically-assigned name using "AS" if s/he chooses.)
>>>
>>> -Chris
>>>
>>>
>>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>>>
>>>> I completely agree. It does start getting confusing.
>> Especially if we
>>>> try to deal with multi field keys.
>>>>
>>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
>>>> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
>>>> USING PigStorage() AS (A, B)
>>>>
>>>> G1 = COGROUP A by (B,C), B by (A, C);
>>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>>>
>>>> What is the schema for G2?
>>>>
>>>> ben
>>>>
>>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>>>> So what is the conclusion here ?
>>>>>
>>>>> group key alias == the first variables group by field ?
>>>>>
>>>>>
>>>>> What happens in a case like this then :
>>>>>
>>>>> --
>>>>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
>>>>> 'somefile2' USING PigStorage() AS (A, C) C = load
>> 'somefile3' USING
>>>>> PigStorage() AS (A, B)
>>>>>
>>>>> G1 = COGROUP A by B, B by A;
>>>>> G2 = COGROUP A by C, C by A;
>>>>> ...
>>>>> --
>>>>>
>>>>> A slightly contrived example for sure, but imo grammer
>> has to be as
>>>>> clearly specified as possible.
>>>>>
>>>>> A reserved keyword as group alias implies we dont hit
>> this problem
>>>>> (group or groupkey or grpkey)... and also the fact that we are
>>>>> backwardly compatible.
>>>>>
>>>>> [I never liked inferred schema prefix section in the schemas doc
>>>>> (which is applied selectively) - makes it extremely tough to
>>>>> generate pig scripts]
>>>>>
>>>>>
>>>>> Regards,
>>>>> Mridul
>>>>>
>>>>> Alan Gates wrote:
>>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
>> used, the
>>>>>> field (or set of fields) that are grouped on are given the alias
>>>>>> 'group'.
>>>>>> This has a couple of issues:
>>>>>>
>>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>>> 2) We don't currently allow 'group' as an alias in an
>> AS. It is
>>>>>> strange to have an alias that can only be assigned by
>> the language
>>>>>> and never by the user.
>>>>>>
>>>>>> Possible solutions:
>>>>>>
>>>>>> I) Status quo. We could fix it so that group is allowed to be
>>>>>> assigned as an alias in AS.
>>>>>>
>>>>>> Pros: Backward compatibility
>>>>>> Cons: a) will make the parser more complicated
>>>>>> b) see 1) above.
>>>>>>
>>>>>>
>>>>>> II) Don't give an implicit alias to the group key(s). If users
>>>>>> want an alias, they can assign it using AS.
>>>>>>
>>>>>> Pros: Simplicity
>>>>>> Cons: We do assign aliases to grouped bags. That is,
>> if we have C
>>>>>> = GROUP B by $0 the resulting schema of C is (group, B).
>> So if we
>>>>>> don't assign an alias to the group key, we now have a
>> schema ($0,
>>>>>> B). This seems strange. And worse yet, if users want
>> to alias the
>>>>>> group key(s), they'll be forced to alias all the grouped bags as
>>>>>> well.
>>>>>>
>>>>>> III) Carry the alias (if any) that the field had before.
>> So if we
>>>>>> had a script like:
>>>>>>
>>>>>> A = load 'myfile' as (x, y, z);
>>>>>> B = group A by x;
>>>>>>
>>>>>> The the schema of B would be (x, A). This is quite natural for
>>>>>> grouping of single columns. But it turns nasty when you
>> group on
>>>>>> multiple columns. Do we then append the names to
>> together? So if
>>>>>> you have
>>>>>>
>>>>>> B = group A by x, y;
>>>>>>
>>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>>
>>>>>> In this case there is also the question of what to do in
>> the case
>>>>>> of cogroups, where the key may be named differently in different
>>>>>> relations.
>>>>>>
>>>>>> A = load 'myfile' as (x, y, z);
>>>>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by x, B by t;
>>>>>>
>>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are
>> both valid?
>>>>>> This
>>>>>> could be resolved by either saying first one always wins, or
>>>>>> allowing either.
>>>>>>
>>>>>> Pros: Very natural for the users, their fields maintain names
>>>>>> through the query.
>>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>>>
>>>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>>>> groupkey or grpkey (or some other suitable choice).
>>>>>> Pros: Least disruptive change. Users only have to go through
>>>>>> their scripts and find places where they use the group alias and
>>>>>> change it to grp (or whatever).
>>>>>> Cons: Still leaves us with a situation where we are assigning a
>>>>>> name to a field arbtrarily, leaving users confused as to
>> how their
>>>>>> fields got named that.
>>>>>>
>>>>>> V) Remove GROUP as a keyword. It is just short for
>> COGROUP of one
>>>>>> relation anyway.
>>>>>>
>>>>>> Pros: Smaller syntax in a language is always good.
>>>>>> Cons: Will break a lot of scripts, and confuse a lot of
>> users who
>>>>>> only think in terms of GROUP and JOIN and never use COGROUP
>>>>>> explicitly.
>>>>>>
>>>>>> One could also conceive of combinations of these. For
>> example, we
>>>>>> always assign a name like grpkey to the group key(s), and in the
>>>>>> single key case we also carry forward the alias that the field
>>>>>> already had, if any.
>>>>>>
>>>>>> Thoughts? Other possibilities?
>>>>>>
>>>>>> Alan.
>>>>
>>>>
>>>
>>> --
>>> Christopher Olston, Ph.D.
>>> Sr. Research Scientist
>>> Yahoo! Research
>>>
>>>
>>>
>>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
RE: Issues with group as an alias
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
What about naming the rest of the fields in the group? Do we want to
continue naming them with the names of the corresponding tables? I think
users find that confusing as well.
Olga
> -----Original Message-----
> From: Alan Gates [mailto:gates@yahoo-inc.com]
> Sent: Monday, June 16, 2008 11:32 AM
> To: pig-dev@incubator.apache.org
> Subject: Re: Issues with group as an alias
>
> I would like to propose a slight modification:
>
> I think that we should continue to support 'group' as the
> alias name for some transition period (3 or maybe 6 months).
> We can remove all references to group as an alias from the
> documentation and print a warning when users use it. But I
> don't think we should drop it immediately, as we'll break
> many scripts.
>
> Other than that I'm fine with the proposal.
>
> Alan.
>
> Chris Olston wrote:
> > No.
> >
> > The standing proposal for Option III is:
> >
> > 1. If you are (CO)Grouping on a *single* field AND in the case of
> > co-group all field names are the same (e.g., cogroup A by url, B by
> > url), then give the group key that name (e.g., "url").
> > 2. Else, do *not* automatically assign any name. The user
> can refer to
> > it as $0 and/or use "AS" to give it a name manually.
> >
> > (To be clear, even in case #1, the user has the option to
> override the
> > automatically-assigned name using "AS" if s/he chooses.)
> >
> > -Chris
> >
> >
> > On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
> >
> >> I completely agree. It does start getting confusing.
> Especially if we
> >> try to deal with multi field keys.
> >>
> >> A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
> >> 'somefile2' USING PigStorage() AS (A, C, Y) C = load 'somefile3'
> >> USING PigStorage() AS (A, B)
> >>
> >> G1 = COGROUP A by (B,C), B by (A, C);
> >> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
> >>
> >> What is the schema for G2?
> >>
> >> ben
> >>
> >> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
> >>> So what is the conclusion here ?
> >>>
> >>> group key alias == the first variables group by field ?
> >>>
> >>>
> >>> What happens in a case like this then :
> >>>
> >>> --
> >>> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
> >>> 'somefile2' USING PigStorage() AS (A, C) C = load
> 'somefile3' USING
> >>> PigStorage() AS (A, B)
> >>>
> >>> G1 = COGROUP A by B, B by A;
> >>> G2 = COGROUP A by C, C by A;
> >>> ...
> >>> --
> >>>
> >>> A slightly contrived example for sure, but imo grammer
> has to be as
> >>> clearly specified as possible.
> >>>
> >>> A reserved keyword as group alias implies we dont hit
> this problem
> >>> (group or groupkey or grpkey)... and also the fact that we are
> >>> backwardly compatible.
> >>>
> >>> [I never liked inferred schema prefix section in the schemas doc
> >>> (which is applied selectively) - makes it extremely tough to
> >>> generate pig scripts]
> >>>
> >>>
> >>> Regards,
> >>> Mridul
> >>>
> >>> Alan Gates wrote:
> >>>> Currently in Pig Latin, anytime a (CO)GROUP statement is
> used, the
> >>>> field (or set of fields) that are grouped on are given the alias
> >>>> 'group'.
> >>>> This has a couple of issues:
> >>>>
> >>>> 1) It's confusing. 'group' is now a keyword and an alias.
> >>>> 2) We don't currently allow 'group' as an alias in an
> AS. It is
> >>>> strange to have an alias that can only be assigned by
> the language
> >>>> and never by the user.
> >>>>
> >>>> Possible solutions:
> >>>>
> >>>> I) Status quo. We could fix it so that group is allowed to be
> >>>> assigned as an alias in AS.
> >>>>
> >>>> Pros: Backward compatibility
> >>>> Cons: a) will make the parser more complicated
> >>>> b) see 1) above.
> >>>>
> >>>>
> >>>> II) Don't give an implicit alias to the group key(s). If users
> >>>> want an alias, they can assign it using AS.
> >>>>
> >>>> Pros: Simplicity
> >>>> Cons: We do assign aliases to grouped bags. That is,
> if we have C
> >>>> = GROUP B by $0 the resulting schema of C is (group, B).
> So if we
> >>>> don't assign an alias to the group key, we now have a
> schema ($0,
> >>>> B). This seems strange. And worse yet, if users want
> to alias the
> >>>> group key(s), they'll be forced to alias all the grouped bags as
> >>>> well.
> >>>>
> >>>> III) Carry the alias (if any) that the field had before.
> So if we
> >>>> had a script like:
> >>>>
> >>>> A = load 'myfile' as (x, y, z);
> >>>> B = group A by x;
> >>>>
> >>>> The the schema of B would be (x, A). This is quite natural for
> >>>> grouping of single columns. But it turns nasty when you
> group on
> >>>> multiple columns. Do we then append the names to
> together? So if
> >>>> you have
> >>>>
> >>>> B = group A by x, y;
> >>>>
> >>>> is the resulting schema (x_y, A)? Ugh.
> >>>>
> >>>> In this case there is also the question of what to do in
> the case
> >>>> of cogroups, where the key may be named differently in different
> >>>> relations.
> >>>>
> >>>> A = load 'myfile' as (x, y, z);
> >>>> B = load 'myotherfile' as (t, u, v); C = cogroup A by x, B by t;
> >>>>
> >>>> Is the resulting schema (x, A, B) or (t, A, B) or are
> both valid?
> >>>> This
> >>>> could be resolved by either saying first one always wins, or
> >>>> allowing either.
> >>>>
> >>>> Pros: Very natural for the users, their fields maintain names
> >>>> through the query.
> >>>> Cons: Quickly gets burdensome in the case of multi-key groups.
> >>>>
> >>>> IV) Assign a non-keyword alias to the group key, like grp or
> >>>> groupkey or grpkey (or some other suitable choice).
> >>>> Pros: Least disruptive change. Users only have to go through
> >>>> their scripts and find places where they use the group alias and
> >>>> change it to grp (or whatever).
> >>>> Cons: Still leaves us with a situation where we are assigning a
> >>>> name to a field arbtrarily, leaving users confused as to
> how their
> >>>> fields got named that.
> >>>>
> >>>> V) Remove GROUP as a keyword. It is just short for
> COGROUP of one
> >>>> relation anyway.
> >>>>
> >>>> Pros: Smaller syntax in a language is always good.
> >>>> Cons: Will break a lot of scripts, and confuse a lot of
> users who
> >>>> only think in terms of GROUP and JOIN and never use COGROUP
> >>>> explicitly.
> >>>>
> >>>> One could also conceive of combinations of these. For
> example, we
> >>>> always assign a name like grpkey to the group key(s), and in the
> >>>> single key case we also carry forward the alias that the field
> >>>> already had, if any.
> >>>>
> >>>> Thoughts? Other possibilities?
> >>>>
> >>>> Alan.
> >>
> >>
> >
> > --
> > Christopher Olston, Ph.D.
> > Sr. Research Scientist
> > Yahoo! Research
> >
> >
> >
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
Good idea.
On Jun 16, 2008, at 11:31 AM, Alan Gates wrote:
> I would like to propose a slight modification:
>
> I think that we should continue to support 'group' as the alias
> name for some transition period (3 or maybe 6 months). We can
> remove all references to group as an alias from the documentation
> and print a warning when users use it. But I don't think we should
> drop it immediately, as we'll break many scripts.
>
> Other than that I'm fine with the proposal.
>
> Alan.
>
> Chris Olston wrote:
>> No.
>>
>> The standing proposal for Option III is:
>>
>> 1. If you are (CO)Grouping on a *single* field AND in the case of
>> co-group all field names are the same (e.g., cogroup A by url, B
>> by url), then give the group key that name (e.g., "url").
>> 2. Else, do *not* automatically assign any name. The user can
>> refer to it as $0 and/or use "AS" to give it a name manually.
>>
>> (To be clear, even in case #1, the user has the option to override
>> the automatically-assigned name using "AS" if s/he chooses.)
>>
>> -Chris
>>
>>
>> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>>
>>> I completely agree. It does start getting confusing. Especially
>>> if we try to
>>> deal with multi field keys.
>>>
>>> A = load 'somefile1' USING PigStorage() AS (B, C, Z)
>>> B = load 'somefile2' USING PigStorage() AS (A, C, Y)
>>> C = load 'somefile3' USING PigStorage() AS (A, B)
>>>
>>> G1 = COGROUP A by (B,C), B by (A, C);
>>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>>
>>> What is the schema for G2?
>>>
>>> ben
>>>
>>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>>> So what is the conclusion here ?
>>>>
>>>> group key alias == the first variables group by field ?
>>>>
>>>>
>>>> What happens in a case like this then :
>>>>
>>>> --
>>>> A = load 'somefile1' USING PigStorage() AS (B, C)
>>>> B = load 'somefile2' USING PigStorage() AS (A, C)
>>>> C = load 'somefile3' USING PigStorage() AS (A, B)
>>>>
>>>> G1 = COGROUP A by B, B by A;
>>>> G2 = COGROUP A by C, C by A;
>>>> ...
>>>> --
>>>>
>>>> A slightly contrived example for sure, but imo grammer has to be as
>>>> clearly specified as possible.
>>>>
>>>> A reserved keyword as group alias implies we dont hit this problem
>>>> (group or groupkey or grpkey)... and also the fact that we are
>>>> backwardly compatible.
>>>>
>>>> [I never liked inferred schema prefix section in the schemas doc
>>>> (which
>>>> is applied selectively) - makes it extremely tough to generate
>>>> pig scripts]
>>>>
>>>>
>>>> Regards,
>>>> Mridul
>>>>
>>>> Alan Gates wrote:
>>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is used,
>>>>> the field
>>>>> (or set of fields) that are grouped on are given the alias
>>>>> 'group'.
>>>>> This has a couple of issues:
>>>>>
>>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>>> 2) We don't currently allow 'group' as an alias in an AS. It is
>>>>> strange to have an alias that can only be assigned by the
>>>>> language and
>>>>> never by the user.
>>>>>
>>>>> Possible solutions:
>>>>>
>>>>> I) Status quo. We could fix it so that group is allowed to be
>>>>> assigned
>>>>> as an alias in AS.
>>>>>
>>>>> Pros: Backward compatibility
>>>>> Cons: a) will make the parser more complicated
>>>>> b) see 1) above.
>>>>>
>>>>>
>>>>> II) Don't give an implicit alias to the group key(s). If users
>>>>> want an
>>>>> alias, they can assign it using AS.
>>>>>
>>>>> Pros: Simplicity
>>>>> Cons: We do assign aliases to grouped bags. That is, if we
>>>>> have C =
>>>>> GROUP B by $0 the resulting schema of C is (group, B). So if
>>>>> we don't
>>>>> assign an alias to the group key, we now have a schema ($0,
>>>>> B). This
>>>>> seems strange. And worse yet, if users want to alias the group
>>>>> key(s),
>>>>> they'll be forced to alias all the grouped bags as well.
>>>>>
>>>>> III) Carry the alias (if any) that the field had before. So if
>>>>> we had a
>>>>> script like:
>>>>>
>>>>> A = load 'myfile' as (x, y, z);
>>>>> B = group A by x;
>>>>>
>>>>> The the schema of B would be (x, A). This is quite natural for
>>>>> grouping
>>>>> of single columns. But it turns nasty when you group on multiple
>>>>> columns. Do we then append the names to together? So if you have
>>>>>
>>>>> B = group A by x, y;
>>>>>
>>>>> is the resulting schema (x_y, A)? Ugh.
>>>>>
>>>>> In this case there is also the question of what to do in the
>>>>> case of
>>>>> cogroups, where the key may be named differently in different
>>>>> relations.
>>>>>
>>>>> A = load 'myfile' as (x, y, z);
>>>>> B = load 'myotherfile' as (t, u, v);
>>>>> C = cogroup A by x, B by t;
>>>>>
>>>>> Is the resulting schema (x, A, B) or (t, A, B) or are both
>>>>> valid? This
>>>>> could be resolved by either saying first one always wins, or
>>>>> allowing
>>>>> either.
>>>>>
>>>>> Pros: Very natural for the users, their fields maintain names
>>>>> through
>>>>> the query.
>>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>>
>>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>>> groupkey or
>>>>> grpkey (or some other suitable choice).
>>>>> Pros: Least disruptive change. Users only have to go through
>>>>> their
>>>>> scripts and find places where they use the group alias and
>>>>> change it to
>>>>> grp (or whatever).
>>>>> Cons: Still leaves us with a situation where we are assigning
>>>>> a name to
>>>>> a field arbtrarily, leaving users confused as to how their
>>>>> fields got
>>>>> named that.
>>>>>
>>>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>>>> relation anyway.
>>>>>
>>>>> Pros: Smaller syntax in a language is always good.
>>>>> Cons: Will break a lot of scripts, and confuse a lot of users
>>>>> who only
>>>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>>>
>>>>> One could also conceive of combinations of these. For example, we
>>>>> always assign a name like grpkey to the group key(s), and in
>>>>> the single
>>>>> key case we also carry forward the alias that the field already
>>>>> had, if
>>>>> any.
>>>>>
>>>>> Thoughts? Other possibilities?
>>>>>
>>>>> Alan.
>>>
>>>
>>
>> --
>> Christopher Olston, Ph.D.
>> Sr. Research Scientist
>> Yahoo! Research
>>
>>
>>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
Re: Issues with group as an alias
Posted by Alan Gates <ga...@yahoo-inc.com>.
I would like to propose a slight modification:
I think that we should continue to support 'group' as the alias name for
some transition period (3 or maybe 6 months). We can remove all
references to group as an alias from the documentation and print a
warning when users use it. But I don't think we should drop it
immediately, as we'll break many scripts.
Other than that I'm fine with the proposal.
Alan.
Chris Olston wrote:
> No.
>
> The standing proposal for Option III is:
>
> 1. If you are (CO)Grouping on a *single* field AND in the case of
> co-group all field names are the same (e.g., cogroup A by url, B by
> url), then give the group key that name (e.g., "url").
> 2. Else, do *not* automatically assign any name. The user can refer to
> it as $0 and/or use "AS" to give it a name manually.
>
> (To be clear, even in case #1, the user has the option to override the
> automatically-assigned name using "AS" if s/he chooses.)
>
> -Chris
>
>
> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>
>> I completely agree. It does start getting confusing. Especially if we
>> try to
>> deal with multi field keys.
>>
>> A = load 'somefile1' USING PigStorage() AS (B, C, Z)
>> B = load 'somefile2' USING PigStorage() AS (A, C, Y)
>> C = load 'somefile3' USING PigStorage() AS (A, B)
>>
>> G1 = COGROUP A by (B,C), B by (A, C);
>> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>>
>> What is the schema for G2?
>>
>> ben
>>
>> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>>> So what is the conclusion here ?
>>>
>>> group key alias == the first variables group by field ?
>>>
>>>
>>> What happens in a case like this then :
>>>
>>> --
>>> A = load 'somefile1' USING PigStorage() AS (B, C)
>>> B = load 'somefile2' USING PigStorage() AS (A, C)
>>> C = load 'somefile3' USING PigStorage() AS (A, B)
>>>
>>> G1 = COGROUP A by B, B by A;
>>> G2 = COGROUP A by C, C by A;
>>> ...
>>> --
>>>
>>> A slightly contrived example for sure, but imo grammer has to be as
>>> clearly specified as possible.
>>>
>>> A reserved keyword as group alias implies we dont hit this problem
>>> (group or groupkey or grpkey)... and also the fact that we are
>>> backwardly compatible.
>>>
>>> [I never liked inferred schema prefix section in the schemas doc (which
>>> is applied selectively) - makes it extremely tough to generate pig
>>> scripts]
>>>
>>>
>>> Regards,
>>> Mridul
>>>
>>> Alan Gates wrote:
>>>> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the
>>>> field
>>>> (or set of fields) that are grouped on are given the alias 'group'.
>>>> This has a couple of issues:
>>>>
>>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>>> 2) We don't currently allow 'group' as an alias in an AS. It is
>>>> strange to have an alias that can only be assigned by the language and
>>>> never by the user.
>>>>
>>>> Possible solutions:
>>>>
>>>> I) Status quo. We could fix it so that group is allowed to be
>>>> assigned
>>>> as an alias in AS.
>>>>
>>>> Pros: Backward compatibility
>>>> Cons: a) will make the parser more complicated
>>>> b) see 1) above.
>>>>
>>>>
>>>> II) Don't give an implicit alias to the group key(s). If users
>>>> want an
>>>> alias, they can assign it using AS.
>>>>
>>>> Pros: Simplicity
>>>> Cons: We do assign aliases to grouped bags. That is, if we have C =
>>>> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
>>>> assign an alias to the group key, we now have a schema ($0, B). This
>>>> seems strange. And worse yet, if users want to alias the group
>>>> key(s),
>>>> they'll be forced to alias all the grouped bags as well.
>>>>
>>>> III) Carry the alias (if any) that the field had before. So if we
>>>> had a
>>>> script like:
>>>>
>>>> A = load 'myfile' as (x, y, z);
>>>> B = group A by x;
>>>>
>>>> The the schema of B would be (x, A). This is quite natural for
>>>> grouping
>>>> of single columns. But it turns nasty when you group on multiple
>>>> columns. Do we then append the names to together? So if you have
>>>>
>>>> B = group A by x, y;
>>>>
>>>> is the resulting schema (x_y, A)? Ugh.
>>>>
>>>> In this case there is also the question of what to do in the case of
>>>> cogroups, where the key may be named differently in different
>>>> relations.
>>>>
>>>> A = load 'myfile' as (x, y, z);
>>>> B = load 'myotherfile' as (t, u, v);
>>>> C = cogroup A by x, B by t;
>>>>
>>>> Is the resulting schema (x, A, B) or (t, A, B) or are both valid?
>>>> This
>>>> could be resolved by either saying first one always wins, or allowing
>>>> either.
>>>>
>>>> Pros: Very natural for the users, their fields maintain names through
>>>> the query.
>>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>>
>>>> IV) Assign a non-keyword alias to the group key, like grp or
>>>> groupkey or
>>>> grpkey (or some other suitable choice).
>>>> Pros: Least disruptive change. Users only have to go through their
>>>> scripts and find places where they use the group alias and change
>>>> it to
>>>> grp (or whatever).
>>>> Cons: Still leaves us with a situation where we are assigning a
>>>> name to
>>>> a field arbtrarily, leaving users confused as to how their fields got
>>>> named that.
>>>>
>>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>>> relation anyway.
>>>>
>>>> Pros: Smaller syntax in a language is always good.
>>>> Cons: Will break a lot of scripts, and confuse a lot of users who
>>>> only
>>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>>
>>>> One could also conceive of combinations of these. For example, we
>>>> always assign a name like grpkey to the group key(s), and in the
>>>> single
>>>> key case we also carry forward the alias that the field already
>>>> had, if
>>>> any.
>>>>
>>>> Thoughts? Other possibilities?
>>>>
>>>> Alan.
>>
>>
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
RE: Issues with group as an alias
Posted by Olga Natkovich <ol...@yahoo-inc.com>.
+1. Makes sense to me
> -----Original Message-----
> From: Chris Olston [mailto:olston@yahoo-inc.com]
> Sent: Monday, June 16, 2008 10:29 AM
> To: pig-dev@incubator.apache.org
> Subject: Re: Issues with group as an alias
>
> No.
>
> The standing proposal for Option III is:
>
> 1. If you are (CO)Grouping on a *single* field AND in the
> case of co- group all field names are the same (e.g., cogroup
> A by url, B by url), then give the group key that name (e.g., "url").
> 2. Else, do *not* automatically assign any name. The user can
> refer to it as $0 and/or use "AS" to give it a name manually.
>
> (To be clear, even in case #1, the user has the option to
> override the automatically-assigned name using "AS" if s/he chooses.)
>
> -Chris
>
>
> On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
>
> > I completely agree. It does start getting confusing.
> Especially if we
> > try to deal with multi field keys.
> >
> > A = load 'somefile1' USING PigStorage() AS (B, C, Z) B = load
> > 'somefile2' USING PigStorage() AS (A, C, Y) C = load
> 'somefile3' USING
> > PigStorage() AS (A, B)
> >
> > G1 = COGROUP A by (B,C), B by (A, C);
> > G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
> >
> > What is the schema for G2?
> >
> > ben
> >
> > On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
> >> So what is the conclusion here ?
> >>
> >> group key alias == the first variables group by field ?
> >>
> >>
> >> What happens in a case like this then :
> >>
> >> --
> >> A = load 'somefile1' USING PigStorage() AS (B, C) B = load
> >> 'somefile2' USING PigStorage() AS (A, C) C = load
> 'somefile3' USING
> >> PigStorage() AS (A, B)
> >>
> >> G1 = COGROUP A by B, B by A;
> >> G2 = COGROUP A by C, C by A;
> >> ...
> >> --
> >>
> >> A slightly contrived example for sure, but imo grammer has
> to be as
> >> clearly specified as possible.
> >>
> >> A reserved keyword as group alias implies we dont hit this problem
> >> (group or groupkey or grpkey)... and also the fact that we are
> >> backwardly compatible.
> >>
> >> [I never liked inferred schema prefix section in the schemas doc
> >> (which is applied selectively) - makes it extremely tough
> to generate
> >> pig scripts]
> >>
> >>
> >> Regards,
> >> Mridul
> >>
> >> Alan Gates wrote:
> >>> Currently in Pig Latin, anytime a (CO)GROUP statement is
> used, the
> >>> field (or set of fields) that are grouped on are given the alias
> >>> 'group'.
> >>> This has a couple of issues:
> >>>
> >>> 1) It's confusing. 'group' is now a keyword and an alias.
> >>> 2) We don't currently allow 'group' as an alias in an AS. It is
> >>> strange to have an alias that can only be assigned by the
> language
> >>> and never by the user.
> >>>
> >>> Possible solutions:
> >>>
> >>> I) Status quo. We could fix it so that group is allowed to be
> >>> assigned as an alias in AS.
> >>>
> >>> Pros: Backward compatibility
> >>> Cons: a) will make the parser more complicated
> >>> b) see 1) above.
> >>>
> >>>
> >>> II) Don't give an implicit alias to the group key(s). If
> users want
> >>> an alias, they can assign it using AS.
> >>>
> >>> Pros: Simplicity
> >>> Cons: We do assign aliases to grouped bags. That is, if
> we have C
> >>> = GROUP B by $0 the resulting schema of C is (group, B).
> So if we
> >>> don't
> >>> assign an alias to the group key, we now have a schema ($0, B).
> >>> This
> >>> seems strange. And worse yet, if users want to alias the group
> >>> key(s), they'll be forced to alias all the grouped bags as well.
> >>>
> >>> III) Carry the alias (if any) that the field had before.
> So if we
> >>> had a script like:
> >>>
> >>> A = load 'myfile' as (x, y, z);
> >>> B = group A by x;
> >>>
> >>> The the schema of B would be (x, A). This is quite natural for
> >>> grouping of single columns. But it turns nasty when you group on
> >>> multiple columns. Do we then append the names to
> together? So if
> >>> you have
> >>>
> >>> B = group A by x, y;
> >>>
> >>> is the resulting schema (x_y, A)? Ugh.
> >>>
> >>> In this case there is also the question of what to do in
> the case of
> >>> cogroups, where the key may be named differently in different
> >>> relations.
> >>>
> >>> A = load 'myfile' as (x, y, z);
> >>> B = load 'myotherfile' as (t, u, v); C = cogroup A by x, B by t;
> >>>
> >>> Is the resulting schema (x, A, B) or (t, A, B) or are
> both valid?
> >>> This could be resolved by either saying first one always wins, or
> >>> allowing either.
> >>>
> >>> Pros: Very natural for the users, their fields maintain names
> >>> through the query.
> >>> Cons: Quickly gets burdensome in the case of multi-key groups.
> >>>
> >>> IV) Assign a non-keyword alias to the group key, like grp or
> >>> groupkey or grpkey (or some other suitable choice).
> >>> Pros: Least disruptive change. Users only have to go
> through their
> >>> scripts and find places where they use the group alias
> and change it
> >>> to grp (or whatever).
> >>> Cons: Still leaves us with a situation where we are assigning a
> >>> name to a field arbtrarily, leaving users confused as to
> how their
> >>> fields got named that.
> >>>
> >>> V) Remove GROUP as a keyword. It is just short for
> COGROUP of one
> >>> relation anyway.
> >>>
> >>> Pros: Smaller syntax in a language is always good.
> >>> Cons: Will break a lot of scripts, and confuse a lot of
> users who
> >>> only think in terms of GROUP and JOIN and never use COGROUP
> >>> explicitly.
> >>>
> >>> One could also conceive of combinations of these. For
> example, we
> >>> always assign a name like grpkey to the group key(s), and in the
> >>> single key case we also carry forward the alias that the field
> >>> already had, if any.
> >>>
> >>> Thoughts? Other possibilities?
> >>>
> >>> Alan.
> >
> >
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>
Re: Issues with group as an alias
Posted by Chris Olston <ol...@yahoo-inc.com>.
No.
The standing proposal for Option III is:
1. If you are (CO)Grouping on a *single* field AND in the case of co-
group all field names are the same (e.g., cogroup A by url, B by
url), then give the group key that name (e.g., "url").
2. Else, do *not* automatically assign any name. The user can refer
to it as $0 and/or use "AS" to give it a name manually.
(To be clear, even in case #1, the user has the option to override
the automatically-assigned name using "AS" if s/he chooses.)
-Chris
On Jun 16, 2008, at 8:25 AM, Benjamin Reed wrote:
> I completely agree. It does start getting confusing. Especially if
> we try to
> deal with multi field keys.
>
> A = load 'somefile1' USING PigStorage() AS (B, C, Z)
> B = load 'somefile2' USING PigStorage() AS (A, C, Y)
> C = load 'somefile3' USING PigStorage() AS (A, B)
>
> G1 = COGROUP A by (B,C), B by (A, C);
> G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
>
> What is the schema for G2?
>
> ben
>
> On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
>> So what is the conclusion here ?
>>
>> group key alias == the first variables group by field ?
>>
>>
>> What happens in a case like this then :
>>
>> --
>> A = load 'somefile1' USING PigStorage() AS (B, C)
>> B = load 'somefile2' USING PigStorage() AS (A, C)
>> C = load 'somefile3' USING PigStorage() AS (A, B)
>>
>> G1 = COGROUP A by B, B by A;
>> G2 = COGROUP A by C, C by A;
>> ...
>> --
>>
>> A slightly contrived example for sure, but imo grammer has to be as
>> clearly specified as possible.
>>
>> A reserved keyword as group alias implies we dont hit this problem
>> (group or groupkey or grpkey)... and also the fact that we are
>> backwardly compatible.
>>
>> [I never liked inferred schema prefix section in the schemas doc
>> (which
>> is applied selectively) - makes it extremely tough to generate pig
>> scripts]
>>
>>
>> Regards,
>> Mridul
>>
>> Alan Gates wrote:
>>> Currently in Pig Latin, anytime a (CO)GROUP statement is used,
>>> the field
>>> (or set of fields) that are grouped on are given the alias 'group'.
>>> This has a couple of issues:
>>>
>>> 1) It's confusing. 'group' is now a keyword and an alias.
>>> 2) We don't currently allow 'group' as an alias in an AS. It is
>>> strange to have an alias that can only be assigned by the
>>> language and
>>> never by the user.
>>>
>>> Possible solutions:
>>>
>>> I) Status quo. We could fix it so that group is allowed to be
>>> assigned
>>> as an alias in AS.
>>>
>>> Pros: Backward compatibility
>>> Cons: a) will make the parser more complicated
>>> b) see 1) above.
>>>
>>>
>>> II) Don't give an implicit alias to the group key(s). If users
>>> want an
>>> alias, they can assign it using AS.
>>>
>>> Pros: Simplicity
>>> Cons: We do assign aliases to grouped bags. That is, if we have
>>> C =
>>> GROUP B by $0 the resulting schema of C is (group, B). So if we
>>> don't
>>> assign an alias to the group key, we now have a schema ($0, B).
>>> This
>>> seems strange. And worse yet, if users want to alias the group
>>> key(s),
>>> they'll be forced to alias all the grouped bags as well.
>>>
>>> III) Carry the alias (if any) that the field had before. So if
>>> we had a
>>> script like:
>>>
>>> A = load 'myfile' as (x, y, z);
>>> B = group A by x;
>>>
>>> The the schema of B would be (x, A). This is quite natural for
>>> grouping
>>> of single columns. But it turns nasty when you group on multiple
>>> columns. Do we then append the names to together? So if you have
>>>
>>> B = group A by x, y;
>>>
>>> is the resulting schema (x_y, A)? Ugh.
>>>
>>> In this case there is also the question of what to do in the case of
>>> cogroups, where the key may be named differently in different
>>> relations.
>>>
>>> A = load 'myfile' as (x, y, z);
>>> B = load 'myotherfile' as (t, u, v);
>>> C = cogroup A by x, B by t;
>>>
>>> Is the resulting schema (x, A, B) or (t, A, B) or are both
>>> valid? This
>>> could be resolved by either saying first one always wins, or
>>> allowing
>>> either.
>>>
>>> Pros: Very natural for the users, their fields maintain names
>>> through
>>> the query.
>>> Cons: Quickly gets burdensome in the case of multi-key groups.
>>>
>>> IV) Assign a non-keyword alias to the group key, like grp or
>>> groupkey or
>>> grpkey (or some other suitable choice).
>>> Pros: Least disruptive change. Users only have to go through their
>>> scripts and find places where they use the group alias and change
>>> it to
>>> grp (or whatever).
>>> Cons: Still leaves us with a situation where we are assigning a
>>> name to
>>> a field arbtrarily, leaving users confused as to how their fields
>>> got
>>> named that.
>>>
>>> V) Remove GROUP as a keyword. It is just short for COGROUP of one
>>> relation anyway.
>>>
>>> Pros: Smaller syntax in a language is always good.
>>> Cons: Will break a lot of scripts, and confuse a lot of users
>>> who only
>>> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>>>
>>> One could also conceive of combinations of these. For example, we
>>> always assign a name like grpkey to the group key(s), and in the
>>> single
>>> key case we also carry forward the alias that the field already
>>> had, if
>>> any.
>>>
>>> Thoughts? Other possibilities?
>>>
>>> Alan.
>
>
--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research
Re: Issues with group as an alias
Posted by Benjamin Reed <br...@yahoo-inc.com>.
I completely agree. It does start getting confusing. Especially if we try to
deal with multi field keys.
A = load 'somefile1' USING PigStorage() AS (B, C, Z)
B = load 'somefile2' USING PigStorage() AS (A, C, Y)
C = load 'somefile3' USING PigStorage() AS (A, B)
G1 = COGROUP A by (B,C), B by (A, C);
G2 = COGROUP G1 by (B_C, A.Z), C by (A, B);
What is the schema for G2?
ben
On Saturday 14 June 2008 06:46:00 Mridul Muralidharan wrote:
> So what is the conclusion here ?
>
> group key alias == the first variables group by field ?
>
>
> What happens in a case like this then :
>
> --
> A = load 'somefile1' USING PigStorage() AS (B, C)
> B = load 'somefile2' USING PigStorage() AS (A, C)
> C = load 'somefile3' USING PigStorage() AS (A, B)
>
> G1 = COGROUP A by B, B by A;
> G2 = COGROUP A by C, C by A;
> ...
> --
>
> A slightly contrived example for sure, but imo grammer has to be as
> clearly specified as possible.
>
> A reserved keyword as group alias implies we dont hit this problem
> (group or groupkey or grpkey)... and also the fact that we are
> backwardly compatible.
>
> [I never liked inferred schema prefix section in the schemas doc (which
> is applied selectively) - makes it extremely tough to generate pig scripts]
>
>
> Regards,
> Mridul
>
> Alan Gates wrote:
> > Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
> > (or set of fields) that are grouped on are given the alias 'group'.
> > This has a couple of issues:
> >
> > 1) It's confusing. 'group' is now a keyword and an alias.
> > 2) We don't currently allow 'group' as an alias in an AS. It is
> > strange to have an alias that can only be assigned by the language and
> > never by the user.
> >
> > Possible solutions:
> >
> > I) Status quo. We could fix it so that group is allowed to be assigned
> > as an alias in AS.
> >
> > Pros: Backward compatibility
> > Cons: a) will make the parser more complicated
> > b) see 1) above.
> >
> >
> > II) Don't give an implicit alias to the group key(s). If users want an
> > alias, they can assign it using AS.
> >
> > Pros: Simplicity
> > Cons: We do assign aliases to grouped bags. That is, if we have C =
> > GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> > assign an alias to the group key, we now have a schema ($0, B). This
> > seems strange. And worse yet, if users want to alias the group key(s),
> > they'll be forced to alias all the grouped bags as well.
> >
> > III) Carry the alias (if any) that the field had before. So if we had a
> > script like:
> >
> > A = load 'myfile' as (x, y, z);
> > B = group A by x;
> >
> > The the schema of B would be (x, A). This is quite natural for grouping
> > of single columns. But it turns nasty when you group on multiple
> > columns. Do we then append the names to together? So if you have
> >
> > B = group A by x, y;
> >
> > is the resulting schema (x_y, A)? Ugh.
> >
> > In this case there is also the question of what to do in the case of
> > cogroups, where the key may be named differently in different relations.
> >
> > A = load 'myfile' as (x, y, z);
> > B = load 'myotherfile' as (t, u, v);
> > C = cogroup A by x, B by t;
> >
> > Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
> > could be resolved by either saying first one always wins, or allowing
> > either.
> >
> > Pros: Very natural for the users, their fields maintain names through
> > the query.
> > Cons: Quickly gets burdensome in the case of multi-key groups.
> >
> > IV) Assign a non-keyword alias to the group key, like grp or groupkey or
> > grpkey (or some other suitable choice).
> > Pros: Least disruptive change. Users only have to go through their
> > scripts and find places where they use the group alias and change it to
> > grp (or whatever).
> > Cons: Still leaves us with a situation where we are assigning a name to
> > a field arbtrarily, leaving users confused as to how their fields got
> > named that.
> >
> > V) Remove GROUP as a keyword. It is just short for COGROUP of one
> > relation anyway.
> >
> > Pros: Smaller syntax in a language is always good.
> > Cons: Will break a lot of scripts, and confuse a lot of users who only
> > think in terms of GROUP and JOIN and never use COGROUP explicitly.
> >
> > One could also conceive of combinations of these. For example, we
> > always assign a name like grpkey to the group key(s), and in the single
> > key case we also carry forward the alias that the field already had, if
> > any.
> >
> > Thoughts? Other possibilities?
> >
> > Alan.
Re: Issues with group as an alias
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
So what is the conclusion here ?
group key alias == the first variables group by field ?
What happens in a case like this then :
--
A = load 'somefile1' USING PigStorage() AS (B, C)
B = load 'somefile2' USING PigStorage() AS (A, C)
C = load 'somefile3' USING PigStorage() AS (A, B)
G1 = COGROUP A by B, B by A;
G2 = COGROUP A by C, C by A;
...
--
A slightly contrived example for sure, but imo grammer has to be as
clearly specified as possible.
A reserved keyword as group alias implies we dont hit this problem
(group or groupkey or grpkey)... and also the fact that we are
backwardly compatible.
[I never liked inferred schema prefix section in the schemas doc (which
is applied selectively) - makes it extremely tough to generate pig scripts]
Regards,
Mridul
Alan Gates wrote:
> Currently in Pig Latin, anytime a (CO)GROUP statement is used, the field
> (or set of fields) that are grouped on are given the alias 'group'.
> This has a couple of issues:
>
> 1) It's confusing. 'group' is now a keyword and an alias.
> 2) We don't currently allow 'group' as an alias in an AS. It is
> strange to have an alias that can only be assigned by the language and
> never by the user.
>
> Possible solutions:
>
> I) Status quo. We could fix it so that group is allowed to be assigned
> as an alias in AS.
>
> Pros: Backward compatibility
> Cons: a) will make the parser more complicated
> b) see 1) above.
>
>
> II) Don't give an implicit alias to the group key(s). If users want an
> alias, they can assign it using AS.
>
> Pros: Simplicity
> Cons: We do assign aliases to grouped bags. That is, if we have C =
> GROUP B by $0 the resulting schema of C is (group, B). So if we don't
> assign an alias to the group key, we now have a schema ($0, B). This
> seems strange. And worse yet, if users want to alias the group key(s),
> they'll be forced to alias all the grouped bags as well.
>
> III) Carry the alias (if any) that the field had before. So if we had a
> script like:
>
> A = load 'myfile' as (x, y, z);
> B = group A by x;
>
> The the schema of B would be (x, A). This is quite natural for grouping
> of single columns. But it turns nasty when you group on multiple
> columns. Do we then append the names to together? So if you have
>
> B = group A by x, y;
>
> is the resulting schema (x_y, A)? Ugh.
>
> In this case there is also the question of what to do in the case of
> cogroups, where the key may be named differently in different relations.
>
> A = load 'myfile' as (x, y, z);
> B = load 'myotherfile' as (t, u, v);
> C = cogroup A by x, B by t;
>
> Is the resulting schema (x, A, B) or (t, A, B) or are both valid? This
> could be resolved by either saying first one always wins, or allowing
> either.
>
> Pros: Very natural for the users, their fields maintain names through
> the query.
> Cons: Quickly gets burdensome in the case of multi-key groups.
>
> IV) Assign a non-keyword alias to the group key, like grp or groupkey or
> grpkey (or some other suitable choice).
> Pros: Least disruptive change. Users only have to go through their
> scripts and find places where they use the group alias and change it to
> grp (or whatever).
> Cons: Still leaves us with a situation where we are assigning a name to
> a field arbtrarily, leaving users confused as to how their fields got
> named that.
>
> V) Remove GROUP as a keyword. It is just short for COGROUP of one
> relation anyway.
>
> Pros: Smaller syntax in a language is always good.
> Cons: Will break a lot of scripts, and confuse a lot of users who only
> think in terms of GROUP and JOIN and never use COGROUP explicitly.
>
> One could also conceive of combinations of these. For example, we
> always assign a name like grpkey to the group key(s), and in the single
> key case we also carry forward the alias that the field already had, if
> any.
>
> Thoughts? Other possibilities?
>
> Alan.