You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Prasanth J (JIRA)" <ji...@apache.org> on 2012/05/28 02:07:22 UTC

[jira] [Created] (PIG-2726) Handling legitimate NULL values

Prasanth J created PIG-2726:
-------------------------------

             Summary: Handling legitimate NULL values
                 Key: PIG-2726
                 URL: https://issues.apache.org/jira/browse/PIG-2726
             Project: Pig
          Issue Type: Sub-task
    Affects Versions: 0.11
            Reporter: Prasanth J
            Assignee: Prasanth J


Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410505#comment-13410505 ] 

Prasanth J commented on PIG-2726:
---------------------------------

Dmitriy, it looks like the Apache License is already granted. I did it when I uploaded the patch. It's showing up for me when I do "Manage Attachements". 
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406072#comment-13406072 ] 

Prasanth J commented on PIG-2726:
---------------------------------

Attaching a git patch. Patch for PIG-2765 is generated on top of this git patch. 
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated PIG-2726:
----------------------------

    Attachment: PIG-2726.1.git.patch
    
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393642#comment-13393642 ] 

Daniel Dai commented on PIG-2726:
---------------------------------

Is there any way to distinguish unknown and string "unknown"?
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated PIG-2726:
----------------------------

    Attachment:     (was: PIG-2726.1.git.patch)
    
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated PIG-2726:
----------------------------

    Attachment: PIG-2726.1.git.patch
    
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393662#comment-13393662 ] 

Prasanth J commented on PIG-2726:
---------------------------------

No. Currently there is no way to distinguish between the "unknown" value in dimensions and "unknown" created by rollups.

I can think of two ways to solve the problem by getting user defined values 
1) add ONNULL "some_value" to cube grammar 
2) use pig property like pig.olap.onnull "some_value" 

these methods allow users to specify a custom value to use for null values in dimensions instead for defaulting to "unknown". Any thoughts?
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398212#comment-13398212 ] 

Prasanth J commented on PIG-2726:
---------------------------------

Dmitriy,

Can you please review this patch? 
Review Board URL: https://reviews.apache.org/r/5470/

Thanks.
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated PIG-2726:
----------------------------

    Attachment: PIG-2726.1.patch
    
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410497#comment-13410497 ] 

Dmitriy V. Ryaboy commented on PIG-2726:
----------------------------------------

I will commit this, and improvements I suggested in RB will come in when PIG-2765 goes in.
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (PIG-2726) Handling legitimate NULL values

Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dmitriy V. Ryaboy resolved PIG-2726.
------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.11

committed to trunk.
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>             Fix For: 0.11
>
>         Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prasanth J updated PIG-2726:
----------------------------

    Attachment: PIG-2726.1.copy.git.patch
    
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Dmitriy V. Ryaboy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410498#comment-13410498 ] 

Dmitriy V. Ryaboy commented on PIG-2726:
----------------------------------------

Prasanth, I need you to reupload the patch and click the apache license button this time...
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Re: [jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Approach as described seems sound to me. Will review path early next week. 

On Jun 13, 2012, at 9:46 PM, "Prasanth J (JIRA)" <ji...@apache.org> wrote:

> 
>    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294806#comment-13294806 ] 
> 
> Prasanth J commented on PIG-2726:
> ---------------------------------
> 
> This patch converts null values in dimensions to "unknown" and uses actual null value for rollup. For a sample input 
> 
> red, ,12
> 
> CUBE operator will now generate
> 
> red,unknown,12
> ,unknown,12
> red, ,12
> , ,12
> 
> Please let me know if it is ok to go ahead with null for rollup? another alternative would be to use "*" or "all" to represent rollup value. 
> 
>> Handling legitimate NULL values
>> -------------------------------
>> 
>>                Key: PIG-2726
>>                URL: https://issues.apache.org/jira/browse/PIG-2726
>>            Project: Pig
>>         Issue Type: Sub-task
>>   Affects Versions: 0.11
>>           Reporter: Prasanth J
>>           Assignee: Prasanth J
>>        Attachments: PIG-2726.1.patch
>> 
>> 
>> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.
> 
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
> For more information on JIRA, see: http://www.atlassian.com/software/jira
> 
> 

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294806#comment-13294806 ] 

Prasanth J commented on PIG-2726:
---------------------------------

This patch converts null values in dimensions to "unknown" and uses actual null value for rollup. For a sample input 

red, ,12

CUBE operator will now generate

red,unknown,12
 ,unknown,12
red, ,12
 , ,12

Please let me know if it is ok to go ahead with null for rollup? another alternative would be to use "*" or "all" to represent rollup value. 
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (PIG-2726) Handling legitimate NULL values

Posted by "Prasanth J (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/PIG-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410518#comment-13410518 ] 

Prasanth J commented on PIG-2726:
---------------------------------

Attached a copy again. I chose apache license option while uploading (in the same way I did before). Please let me know if its still not visible. The license logo doesn't show up in the JIRA. But it shows up when I do manage attachments.
                
> Handling legitimate NULL values
> -------------------------------
>
>                 Key: PIG-2726
>                 URL: https://issues.apache.org/jira/browse/PIG-2726
>             Project: Pig
>          Issue Type: Sub-task
>    Affects Versions: 0.11
>            Reporter: Prasanth J
>            Assignee: Prasanth J
>         Attachments: PIG-2726.1.copy.git.patch, PIG-2726.1.git.patch, PIG-2726.1.patch
>
>
> Look into SQL/Oracle server for how they are handling legitimate NULL values in the input while performing operations like roll-up, filtering etc. Current implementation outputs NULL string which should be replaced by actual null value.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira