You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Prasanth J (JIRA)" <ji...@apache.org> on 2012/05/28 02:07:22 UTC
[jira] [Created] (PIG-2726) Handling legitimate NULL values
Prasanth J created PIG-2726:
-------------------------------
Summary: Handling legitimate NULL values
Key: PIG-2726
URL: https://issues.apache.org/jira/browse/PIG-2726
Project: Pig
Issue Type: Sub-task
Affects Versions: 0.11
Reporter: Prasanth J
Assignee: Prasanth J
Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410505#comment-13410505 ]
Prasanth J commented on PIG-2726:
---------------------------------
Dmitriy, it looks like the Apache License is already granted. I did it when I uploaded the patch. It's showing up for me when I do "Manage Attachements".
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406072#comment-13406072 ]
Prasanth J commented on PIG-2726:
---------------------------------
Attaching a git patch. Patch for PIG-2765 is generated on top of this git patch.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth J updated PIG-2726:
----------------------------
Attachment: PIG-2726.1.git.patch
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393642#comment-13393642 ]
Daniel Dai commented on PIG-2726:
---------------------------------
Is there any way to distinguish unknown and string "unknown"?
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth J updated PIG-2726:
----------------------------
Attachment: (was: PIG-2726.1.git.patch)
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth J updated PIG-2726:
----------------------------
Attachment: PIG-2726.1.git.patch
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393662#comment-13393662 ]
Prasanth J commented on PIG-2726:
---------------------------------
No. Currently there is no way to distinguish between the "unknown" value in dimensions and "unknown" created by rollups.
I can think of two ways to solve the problem by getting user defined values
1) add ONNULL "some_value" to cube grammar
2) use pig property like pig.olap.onnull "some_value"
these methods allow users to specify a custom value to use for null values in dimensions instead for defaulting to "unknown". Any thoughts?
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398212#comment-13398212 ]
Prasanth J commented on PIG-2726:
---------------------------------
Dmitriy,
Can you please review this patch?
Review Board URL: https://reviews.apache.org/r/5470/
Thanks.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth J updated PIG-2726:
----------------------------
Attachment: PIG-2726.1.patch
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410497#comment-13410497 ]
Dmitriy V. Ryaboy commented on PIG-2726:
----------------------------------------
I will commit this, and improvements I suggested in RB will come in when PIG-2765 goes in.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (PIG-2726) Handling legitimate NULL values
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitriy V. Ryaboy resolved PIG-2726.
------------------------------------
Resolution: Fixed
Fix Version/s: 0.11
committed to trunk.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Fix For: 0.11
>
> Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prasanth J updated PIG-2726:
----------------------------
Attachment: PIG-2726.1.copy.git.patch
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410498#comment-13410498 ]
Dmitriy V. Ryaboy commented on PIG-2726:
----------------------------------------
Prasanth, I need you to reupload the patch and click the apache license button this time...
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Approach as described seems sound to me. Will review path early next week.
On Jun 13, 2012, at 9:46 PM, "Prasanth J (JIRA)" <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294806#comment-13294806 ]
>
> Prasanth J commented on PIG-2726:
> ---------------------------------
>
> This patch converts null values in dimensions to "unknown" and uses actual null value for rollup. For a sample input
>
> red, ,12
>
> CUBE operator will now generate
>
> red,unknown,12
> ,unknown,12
> red, ,12
> , ,12
>
> Please let me know if it is ok to go ahead with null for rollup? another alternative would be to use "*" or "all" to represent rollup value.
>
>> Handling legitimate NULL values
>> -------------------------------
>>
>> Key: PIG-2726
>> URL: https://issues.apache.org/jira/browse/PIG-2726
>> Project: Pig
>> Issue Type: Sub-task
>> Affects Versions: 0.11
>> Reporter: Prasanth J
>> Assignee: Prasanth J
>> Attachments: PIG-2726.1.patch
>>
>>
>> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294806#comment-13294806 ]
Prasanth J commented on PIG-2726:
---------------------------------
This patch converts null values in dimensions to "unknown" and uses actual null value for rollup. For a sample input
red, ,12
CUBE operator will now generate
red,unknown,12
,unknown,12
red, ,12
, ,12
Please let me know if it is ok to go ahead with null for rollup? another alternative would be to use "*" or "all" to represent rollup value.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2726) Handling legitimate NULL values
Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410518#comment-13410518 ]
Prasanth J commented on PIG-2726:
---------------------------------
Attached a copy again. I chose apache license option while uploading (in the same way I did before). Please let me know if its still not visible. The license logo doesn't show up in the JIRA. But it shows up when I do manage attachments.
> Handling legitimate NULL values
> -------------------------------
>
> Key: PIG-2726
> URL: https://issues.apache.org/jira/browse/PIG-2726
> Project: Pig
> Issue Type: Sub-task
> Affects Versions: 0.11
> Reporter: Prasanth J
> Assignee: Prasanth J
> Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira