Flattening bags and tuples without a known schema

[Santhosh Srinivasan <sm...@yahoo-inc.com>]

This email discusses a use case of flattening a bag or tuple when the
schema of the bag or tuple is not known, i.e., null.

When UDFs return bags or tuples (complex type), the schema of the
complex type can be declared via the outputSchema method of the UDF. By
default, the outputSchema method in EvalFunc (the abstract base class)
returns null. When users try to flatten the output of the UDF, the
schema of the flattened column cannot be determined. An example follows.

E.g.: 

--myudf returns a bag whose schema is null, i.e., not declared
B = foreach A generate flatten(myudf), $1 as x;

In the above example, since the schema of the bag returned by myudf is
not known, we have two possible options:

1. Erring on the side of safety, set the schema of the flattened column
to be a bytearray. While this is a safe assumption, authors of the UDF
who are aware of the exact return value of the UDF, will try to access
the elements appropriately. For example, if myudf returned a bag with
tuples containing 3 elements, the following might be a possible use
case:

C = foreach B generate $2 as mycolumn;

At this point, the safe assumption about the flattened column being a
single column of type bytearray will generate {bytearray, x: bytearray}
as the schema for B. As a result, statement C will generate a parse
exception for out of bound access.

Given the fact that UDF authors have complete knowledge about the return
values of the UDF, they should override the outputSchema method in the
UDF to ensure correct schemas. The other option is to specify the schema
as part of the "AS" clause in the generate statement, i.e.,

B = foreach A generate flatten(myudf) as (name: chararray, age: int,
gpa: float), $1 as x;

2. Set the schema of the foreach to be unknown or null. The bag returned
by the UDF could contain arbitrary number of columns, making it
impossible to set the correct column number for the other expression, x
in the generate clause. In all likelihood, this will break existing pig
scripts as:

B = foreach A generate flatten(myudf), $1 as x;
C = foreach B generate $1 + x;


Currently, I have an implementation for option 1. Any
thoughts/suggestions/comments are welcome.

Thanks,
Santhosh

Re: Flattening bags and tuples without a known schema

[Alan Gates <ga...@yahoo-inc.com>]

I vote for option 2, as it is consistent with other pig operations.  
When we load a file and no schema is given, we make no assumptions.  
When we union two relations with differing schemas, the resulting 
relation has no schema.  I think it makes sense to do the same thing 
here.  If the user happens to know his UDF's schema, he can provide it 
via an AS clause.  I agree that this will break some scripts but it is 
consistent with the rest of the way we do things.

Alan.

Santhosh Srinivasan wrote:
> This email discusses a use case of flattening a bag or tuple when the
> schema of the bag or tuple is not known, i.e., null.
>
> When UDFs return bags or tuples (complex type), the schema of the
> complex type can be declared via the outputSchema method of the UDF. By
> default, the outputSchema method in EvalFunc (the abstract base class)
> returns null. When users try to flatten the output of the UDF, the
> schema of the flattened column cannot be determined. An example follows.
>
> E.g.: 
>
> --myudf returns a bag whose schema is null, i.e., not declared
> B = foreach A generate flatten(myudf), $1 as x;
>
> In the above example, since the schema of the bag returned by myudf is
> not known, we have two possible options:
>
> 1. Erring on the side of safety, set the schema of the flattened column
> to be a bytearray. While this is a safe assumption, authors of the UDF
> who are aware of the exact return value of the UDF, will try to access
> the elements appropriately. For example, if myudf returned a bag with
> tuples containing 3 elements, the following might be a possible use
> case:
>
> C = foreach B generate $2 as mycolumn;
>
> At this point, the safe assumption about the flattened column being a
> single column of type bytearray will generate {bytearray, x: bytearray}
> as the schema for B. As a result, statement C will generate a parse
> exception for out of bound access.
>
> Given the fact that UDF authors have complete knowledge about the return
> values of the UDF, they should override the outputSchema method in the
> UDF to ensure correct schemas. The other option is to specify the schema
> as part of the "AS" clause in the generate statement, i.e.,
>
> B = foreach A generate flatten(myudf) as (name: chararray, age: int,
> gpa: float), $1 as x;
>
> 2. Set the schema of the foreach to be unknown or null. The bag returned
> by the UDF could contain arbitrary number of columns, making it
> impossible to set the correct column number for the other expression, x
> in the generate clause. In all likelihood, this will break existing pig
> scripts as:
>
> B = foreach A generate flatten(myudf), $1 as x;
> C = foreach B generate $1 + x;
>
>
> Currently, I have an implementation for option 1. Any
> thoughts/suggestions/comments are welcome.
>
> Thanks,
> Santhosh
>