You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Andy Schlaikjer (JIRA)" <ji...@apache.org> on 2012/05/22 02:21:41 UTC
[jira] [Created] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Andy Schlaikjer created PIG-2715:
------------------------------------
Summary: Cannot specify schema field alias for map value via piglatin
Key: PIG-2715
URL: https://issues.apache.org/jira/browse/PIG-2715
Project: Pig
Issue Type: Bug
Components: parser
Affects Versions: 0.10.0
Reporter: Andy Schlaikjer
Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
{code}
%default INPUT_PATH 'map_value_alias_bug.tsv';
A = LOAD '$INPUT_PATH' AS (
id: long,
-- events: [count: int] -- parse failure
events: [int]
);
DESCRIBE A;
A = FOREACH A GENERATE
id,
-- (map[count: int]) events -- parse failure
-- ([count: int]) events -- parse failure
-- events: [count: int] -- parse failure
-- events AS [count: int] -- parse failure
-- events AS events: [count: int] -- parse failure
([int]) events -- this is okay
;
DESCRIBE A;
quit;
{code}
When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
{code}
message: {id: long,events: map[events: int])}}
{code}
This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281314#comment-13281314 ]
Daniel Dai commented on PIG-2715:
---------------------------------
What do we need alias for? I cannot think of a use case for that.
When you print out the schema, there is no alias: "map[int]". And we will not parse "map[count:int]".
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Jonathan Coveney (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281298#comment-13281298 ]
Jonathan Coveney commented on PIG-2715:
---------------------------------------
Daniel, I am confused. In many cases, users (or functions) can specify the schema of the value in the map. As above, it's saying that the key is a String (which is implicit as this is the only acceptable key type), and the value is an int (named events, though that doesn't mean much).
Why wouldn't we allow this? And if it isn't allowed, we shouldn't print it in the toString, it shouldn't parse, etc. It's very confusing to have. But I think we should allow it, and it should mean the same thing all schema's mean in pig: the values should be of this expected type.
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Andy Schlaikjer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281256#comment-13281256 ]
Andy Schlaikjer commented on PIG-2715:
--------------------------------------
I agree, though I also feel that it's too easy for java code to create an output (or load) schema which specifies a field alias for map values. This in turn leads to problems when strict matching of schemas is required and clients are unable to affect the alias via piglatin.
So, perhaps {{FieldSchema(String a, Schema s, byte t)}} could be hardened to prohibit this:
http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/impl/logicalLayer/schema/Schema.java?view=markup#l131
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281331#comment-13281331 ]
Daniel Dai commented on PIG-2715:
---------------------------------
That's not good. We should definitely remove the alias in the describe.
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281244#comment-13281244 ]
Daniel Dai commented on PIG-2715:
---------------------------------
It is by design, map is a collection of key-value pair, there is no notion of field alias. What do you need field alias for?
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281277#comment-13281277 ]
Daniel Dai commented on PIG-2715:
---------------------------------
Yes, alias name for map value is meaningless. We shall clearly document it.
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2715) Cannot specify schema field alias for
map value via piglatin
Posted by "Andy Schlaikjer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281319#comment-13281319 ]
Andy Schlaikjer commented on PIG-2715:
--------------------------------------
> When you print out the schema, there is no alias: "map[int]". And we will not parse "map[count:int]".
When I "DESCRIBE relation_with_map_with_value_alias" I *do* see map value alias e.g. "map[count: int]" when code has programmatically set the alias, in this case within a LoadFunc implementation.
> Cannot specify schema field alias for map value via piglatin
> ------------------------------------------------------------
>
> Key: PIG-2715
> URL: https://issues.apache.org/jira/browse/PIG-2715
> Project: Pig
> Issue Type: Bug
> Components: parser
> Affects Versions: 0.10.0
> Reporter: Andy Schlaikjer
>
> Programmatically, pig components are able to define schema field aliases for map values, but the same cannot be done via piglatin. For example:
> {code}
> %default INPUT_PATH 'map_value_alias_bug.tsv';
> A = LOAD '$INPUT_PATH' AS (
> id: long,
> -- events: [count: int] -- parse failure
> events: [int]
> );
> DESCRIBE A;
> A = FOREACH A GENERATE
> id,
> -- (map[count: int]) events -- parse failure
> -- ([count: int]) events -- parse failure
> -- events: [count: int] -- parse failure
> -- events AS [count: int] -- parse failure
> -- events AS events: [count: int] -- parse failure
> ([int]) events -- this is okay
> ;
> DESCRIBE A;
> quit;
> {code}
> When using the [Elephant Bird project|https://github.com/kevinweil/elephant-bird/]'s [ThriftToPig utility|https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/util/ThriftToPig.java#L273] along with a Thrift message spec containing a map field, I see schemas of the following form:
> {code}
> message: {id: long,events: map[events: int])}}
> {code}
> This same DESCRIBE output does not parse if used within piglatin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira