You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Scott Gorlin <sg...@choicestream.com> on 2011/12/13 17:47:25 UTC

empty tuple () vs (null)

Hi,

I noticed the following as I'm learning Pig, which I thought I'd share to get some insight on.  It seems to me that Pig cannot (automatically) distinguish between an empty tuple and a tuple with a single null value.  Example:

data = LOAD 'testusers.dat' as (user, empty:tuple(), partial:tuple(p));

DUMP data;
(1,(),())
(2,(),(4))
(3,(),(5))

more = FOREACH data GENERATE user, empty, partial, (), (null), (null,null);

DESCRIBE more;
more: {user: bytearray,empty: (),partial: (p: bytearray),(),(bytearray),(bytearray,bytearray)}

ILLUSTRATE more;
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| more     | user:bytearray    | empty:tuple()    | partial:tuple(p:bytearray)       | :tuple()    | :tuple(:bytearray)       | :tuple(:bytearray,:bytearray)          |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|          | 1                 | ()               | ()                               | ()          | ()                       | (, )                                   |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------

DUMP more;
(1,(),(),(),(),(,))
(2,(),(4),(),(),(,))
(3,(),(5),(),(),(,))

STORE more INTO 'out';
1 () () () () (,)
2 () (4) () () (,)
3 () (5) () () (,)

(Pig 0.9.1 in local mode).

It seems to me that an empty tuple and a tuple with a single null value are both represented as ().  It also seems like explicitly declaring the schema allows a Pig programmer to interpret () in either manner, which is good.

Interestingly, if 'partial' is instead defined as 'partial:tuple(),' then Pig can still handle the data; I guess it automatically adds fields to tuple when it encounters them in each row.

Thoughts?



________________________________
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers.