You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Thejas M Nair (JIRA)" <ji...@apache.org> on 2011/04/29 23:57:03 UTC
[jira] [Created] (PIG-2023) lineage tracking for casting should
compare LoadCaster returned from LoadFunc instead of comparing the FuncSpec
lineage tracking for casting should compare LoadCaster returned from LoadFunc instead of comparing the FuncSpec
----------------------------------------------------------------------------------------------------------------
Key: PIG-2023
URL: https://issues.apache.org/jira/browse/PIG-2023
Project: Pig
Issue Type: Improvement
Reporter: Thejas M Nair
Fix For: 0.10
When lineage of a column is tracked for the purpose of finding the LoadCaster associated with a column, and it finds that a column has two possible sources, it associates a LoadCaster (through a LoadFunc) only if the funcspec for LoadFunc in both cases are the same. But it is possible that the two LoadFunc with different func spec actually use the same LoadCaster (for example the default of Utf8StorageConverter). If the LoadFunc funcspec don't match, the LoadCaster returned by the LoadFunc should also be compred. If they are equal, this LoadCaster should be associated with the column . The LoadCaster implementation would need to override equals().
For example, in this case the columns in relation u use the same LoadCaster -
{code}
l1 = load 'x' using PigStorage(',') as (a,b);
l2 = load 'y' using PigStorage(':') as (a,b);
u = union l1,l2;
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2023) lineage tracking for casting should
compare LoadCaster returned from LoadFunc instead of comparing the FuncSpec
Posted by "Olga Natkovich (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2023:
--------------------------------
Fix Version/s: (was: 0.10)
> lineage tracking for casting should compare LoadCaster returned from LoadFunc instead of comparing the FuncSpec
> ----------------------------------------------------------------------------------------------------------------
>
> Key: PIG-2023
> URL: https://issues.apache.org/jira/browse/PIG-2023
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
>
> When lineage of a column is tracked for the purpose of finding the LoadCaster associated with a column, and it finds that a column has two possible sources, it associates a LoadCaster (through a LoadFunc) only if the funcspec for LoadFunc in both cases are the same. But it is possible that the two LoadFunc with different func spec actually use the same LoadCaster (for example the default of Utf8StorageConverter). If the LoadFunc funcspec don't match, the LoadCaster returned by the LoadFunc should also be compred. If they are equal, this LoadCaster should be associated with the column . The LoadCaster implementation would need to override equals().
> For example, in this case the columns in relation u use the same LoadCaster -
> {code}
> l1 = load 'x' using PigStorage(',') as (a,b);
> l2 = load 'y' using PigStorage(':') as (a,b);
> u = union l1,l2;
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira