[jira] [Commented] (PIG-5201) Null handling on FLATTEN

["Koji Noguchi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-5201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945728#comment-15945728 ] 

Koji Noguchi commented on PIG-5201:
-----------------------------------

{code:title=test.pig}
A = load 'input.txt' as (a1:bag {(a1_1:chararray, a1_2:chararray)}, a2:bag{(a2_1:chararray, a2_2:chararray)}, a3:chararray);
B = FOREACH A generate flatten(a1), flatten(a2), a3;
DUMP B;
{code}

Input
||a1 || a2 || a3 ||
| {(1,2)} | | 1 |
| {(2,3),(3,4)} | {(a,b),(c,d)} | 2 |
| | | 3 |
| | {(e,f)} | 4 |
| {,(7,8)} | {(g,h),(i,j)} | 5 |
Output
{noformat}
(1,2,,1)
(2,3,a,b,2)
(2,3,c,d,2)
(3,4,a,b,2)
(3,4,c,d,2)
(,,3)
(,e,f,4)
(7,8,g,h,5)
(7,8,i,j,5)
{noformat}

Formatting the output in a table
||a2_1|| a2_2 || a3_1 || a3_2 || a4 ||
| 1 | 2 |  | 1 |
| 2 | 3 | a | b | 2 |
| 2 | 3 | c | d | 2 |
| 3 | 4 | a | b | 2 |
| 3 | 4 | c | d | 2 |
|  |  | 3 |
|  | e | f | 4 |
| 7 | 8 | g | h | 5 |
| 7 | 8 | i | j | 5 |

Using different loader({{mock.Storage}}),  I can also see null and \{(g,h),(i,j)\} pair creating 
||a2_1|| a2_2 || a3_1 || a3_2 || a4 ||
| | g | h | 5 | 
| | i | j | 5 |

A couple of quetions.

(1) What should be the behavior for FLATTEN(null)?   For FLATTEN(empty_bag), we drop them.  Shall we do the same?  (Offline, Rohini said no.)

(2) What should be the behavior for null record within the bag in FLATTEN(bag-with-null) ?  

and probably a separate jira, but 
(3) Why is PigStorage not showing the empty item for \{,(7,8)\} ?

> Null handling on FLATTEN
> ------------------------
>
>                 Key: PIG-5201
>                 URL: https://issues.apache.org/jira/browse/PIG-5201
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Minor
>
> Sometimes, FLATTEN(null) or FLATTEN(bag-with-null) seem to produce incorrect results.
> Test code/script to follow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)