You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org> on 2012/05/09 20:21:50 UTC
[jira] [Created] (PIG-2691) Duplicate TOKENIZE schema
Gianmarco De Francisci Morales created PIG-2691:
---------------------------------------------------
Summary: Duplicate TOKENIZE schema
Key: PIG-2691
URL: https://issues.apache.org/jira/browse/PIG-2691
Project: Pig
Issue Type: Bug
Reporter: Gianmarco De Francisci Morales
TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
We could paramenterize the schema on the name of the field being tokenized.
{code}
grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
<line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
grunt> describe e
e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282099#comment-13282099 ]
Daniel Dai commented on PIG-2691:
---------------------------------
Patch looks good.
One potential issue is it introduces some backward compatibility. The output schema name for TOKENIZE change. If anyone rely on it, he/she has to change the script. Is that fine or do we need a flag?
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai reassigned PIG-2691:
-------------------------------
Assignee: Jie Li
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282066#comment-13282066 ]
Jie Li commented on PIG-2691:
-----------------------------
Oops, broke some unit tests. Fixing.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Labels: simple
> Attachments: PIG-2691.patch
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2691:
------------------------
Attachment: PIG-2691.patch
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Labels: simple
> Attachments: PIG-2691.patch
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282128#comment-13282128 ]
Jie Li commented on PIG-2691:
-----------------------------
As there was no documentation on the field schema of TOKENIZE, can we assume that if users want to use the field name, she would explicitly name it by AS? If so, then this change wouldn't break the script.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2691:
------------------------
Status: Patch Available (was: Open)
Changed the field alias of the TOKENIZE result from "bag_of_tokenTuples" to "bag_of_tokenTuples_from_{original_alias}". For example, the result alias of TOKENIZE(a) would be bag_of_tokenTuples_from_a.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Labels: simple
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282195#comment-13282195 ]
Daniel Dai commented on PIG-2691:
---------------------------------
Sounds good. +1
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284767#comment-13284767 ]
Gianmarco De Francisci Morales commented on PIG-2691:
-----------------------------------------------------
Got some failures during testing, but I am not sure about the sanity of my environment.
I am rerunning them in a different one.
Sorry for the delay.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gianmarco De Francisci Morales updated PIG-2691:
------------------------------------------------
Resolution: Fixed
Fix Version/s: 0.11
Release Note: TOKENIZE: the default name of the field in the schema produced by this UDF now depends on the input field. This change could break your script if you were relying on the field being called "bag_of_tokenTuples" (i.e. you were not using an AS clause to rename the field).
Hadoop Flags: Incompatible change
Status: Resolved (was: Patch Available)
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Fix For: 0.11
>
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2691:
------------------------
Attachment: PIG-2691.patch.2
Fixed one unit test for TOKENIZE in TestLogicalPlanBuilder, and added another unit test using the query in the description.
Passed all the other unit tests of test-commit.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2691) Duplicate TOKENIZE schema
Posted by "Gianmarco De Francisci Morales (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282184#comment-13282184 ]
Gianmarco De Francisci Morales commented on PIG-2691:
-----------------------------------------------------
I agree with Jie, though I can imagine people are anyway using it somewhere. We should add something like this to the release notes:
TOKENIZE: the default name of the field in the schema produced by this UDF now depends on the input field. This change could break your script if you were relying on the field being called "bag_of_tokenTuples" (i.e. you were not using an AS clause to rename the field).
Apart from that, I would commit once tests pass.
> Duplicate TOKENIZE schema
> -------------------------
>
> Key: PIG-2691
> URL: https://issues.apache.org/jira/browse/PIG-2691
> Project: Pig
> Issue Type: Bug
> Reporter: Gianmarco De Francisci Morales
> Assignee: Jie Li
> Labels: simple
> Attachments: PIG-2691.patch, PIG-2691.patch.2
>
>
> TOKENIZE produces a fixed named schema that results in duplicates if used more than once in the same generate statement.
> We could paramenterize the schema on the name of the field being tokenized.
> {code}
> grunt> q = LOAD 'file' AS (source:chararray, target:chararray);
> grunt> e = FOREACH q GENERATE TOKENIZE(source), TOKENIZE(target);
> 2012-05-09 20:18:37,235 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1108:
> <line 2, column 14> Duplicate schema alias: bag_of_tokenTuples
> grunt> e = FOREACH q GENERATE TOKENIZE(source) as s_entities, TOKENIZE(target) as t_entities;
> grunt> describe e
> e: {s_entities: {tuple_of_tokens: (token: chararray)},t_entities: {tuple_of_tokens: (token: chararray)}}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira