You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Chang Luo (JIRA)" <ji...@apache.org> on 2012/11/01 05:09:13 UTC
[jira] [Created] (PIG-3021) Split results missing records when
there is null values in the column comparison
Chang Luo created PIG-3021:
------------------------------
Summary: Split results missing records when there is null values in the column comparison
Key: PIG-3021
URL: https://issues.apache.org/jira/browse/PIG-3021
Project: Pig
Issue Type: Bug
Affects Versions: 0.10.0
Reporter: Chang Luo
Suppose a(x, y)
split a into b if x==y, c otherwise;
One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
To workaround this, I have to change to the following:
split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499616#comment-13499616 ]
Cheolsoo Park commented on PIG-3021:
------------------------------------
Hi Chang,
I won't close this jira if you disagree. :-) We can definitely add an example to the SPLIT doc, and you're welcome to submit a patch for that if you like to. I will be happy to review/commit it.
{quote}
I can't post to the users mailing list. Could you please forward my comments to that list?
{quote}
You can subscribe user@pig.apache.org. You're welcome to open a discussion on this matter on the mailing list.
http://pig.apache.org/mailing_lists.html
Thanks!
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Chang Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488889#comment-13488889 ]
Chang Luo commented on PIG-3021:
--------------------------------
Hi Cheolsoo,
Thanks for the quick response. NULL is very tricky as in the SQL world. I can understand your arguments for expected behavior. However, I don't think this semantic is intuitive. When a user use OTHERWISE to define default collection, he expects the default collection is a "catch all" collection.
In the above example, if x or y is null, it will fail the condition x==y and users expect it goes to the default collection defined by OTHERWISE.
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499501#comment-13499501 ]
Cheolsoo Park commented on PIG-3021:
------------------------------------
Given the discussion on the user mailing list, I don't think that we're going to change the current behavior:
http://search-hadoop.com/m/5KtFWXN4me1/CONCAT%2528null%252C+%2522something%2522%2529+%253D%253D+NULL+%253F&subj=CONCAT+null+something+NULL+
I am going to close this jira as a 'won't fix'. Please let me if anyone has objections.
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488901#comment-13488901 ]
Cheolsoo Park commented on PIG-3021:
------------------------------------
Hi Chang,
I think that the current behavior of OTHERWISE is from that we translate it to a "FILTER BY NOT (expr)", and I am open to discussion about whether this is good or bad.
But it is tricky to to change this because it is backward incompatible. Some people may rely on the current behavior, so if we change semantics, their scripts have to be modified. Therefore, we should change it only if there are more benefits than inconveniences by introducing a backward incompatibility.
Nevertheless, I am happy to listen to other opinions.
Thanks!
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Cheolsoo Park (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488787#comment-13488787 ]
Cheolsoo Park commented on PIG-3021:
------------------------------------
Hi Chang,
Thank you very much for raising an issue.
I understand your argument, but that's an expected behavior. Given your example, here are the plans built by Pig:
{code:title=EXPLAIN b}
b: (Name: LOStore Schema: x#9:int,y#10:int)
|
|---b: (Name: LOSplitOutput Schema: x#9:int,y#10:int)
| |
| (Name: Equal Type: boolean Uid: 8)
| |
| |---x:(Name: Project Type: int Uid: 3 Input: 0 Column: 0)
| |
| |---y:(Name: Project Type: int Uid: 4 Input: 0 Column: 1)
|
|---1-4: (Name: LOSplit Schema: x#3:int,y#4:int)
...
{code}
{code:title=EXPLAIN c}
c: (Name: LOStore Schema: x#21:int,y#22:int)
|
|---c: (Name: LOSplitOutput Schema: x#21:int,y#22:int)
| |
| (Name: Not Type: boolean Uid: 20)
| |
| |---(Name: Equal Type: boolean Uid: 19)
| |
| |---x:(Name: Project Type: int Uid: 13 Input: 0 Column: 0)
| |
| |---y:(Name: Project Type: int Uid: 14 Input: 0 Column: 1)
|
|---1-7: (Name: LOSplit Schema: x#13:int,y#14:int)b: (Name: LOStore Schema: x#9:int,y#10:int)
...
{code}
As can be seen, b and c are filtered by expressions == and !=, and the Pig manual says the following [1]:
- Comparison operators: If either subexpression is null, the result is null.
- FILTER operator: If a filter expression results in null value, the filter does not pass them through
So if either x or y is null, they will be dropped.
In addition, the Pig manual also says [2]:
- SPLIT operator: A tuple may not be assigned to any relation.
Does this make sense?
[1] http://pig.apache.org/docs/r0.10.0/basic.html#nulls
[2] http://pig.apache.org/docs/r0.10.0/basic.html#SPLIT
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3021) Split results missing records when
there is null values in the column comparison
Posted by "Chang Luo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499605#comment-13499605 ]
Chang Luo commented on PIG-3021:
--------------------------------
I agree with the discussion about NULL. I also agreed CONCAT(null, something) = null. My issue is on the OTHERWISE keyword not NULL. Maybe it only has issues when NULL is involved but maybe there might be other cases where the same confusion occurs.
I understand it's breaking change and ok with won't-fix. But if I were to redesign SPLIT/OTHERWISE from beginning, I would make OTHERWISE a catch-all default collection.
One suggestion is to give an explicit example of combination of NULL and OTHERWISE keywords in the SPLIT documentation.
I can't post to the users mailing list. Could you please forward my comments to that list? Thanks!
> Split results missing records when there is null values in the column comparison
> --------------------------------------------------------------------------------
>
> Key: PIG-3021
> URL: https://issues.apache.org/jira/browse/PIG-3021
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Chang Luo
>
> Suppose a(x, y)
> split a into b if x==y, c otherwise;
> One will expect the union of b and c will be a. However, if x or y is null, the record won't appear in either b or c.
> To workaround this, I have to change to the following:
> split a into b if x is not null and y is not null and x==y, c otherwise;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira