Is automatic type conversion only kicked in when your loadFunc doesn't implement LoadMetadata?

[Jae Lee <Ja...@forward.co.uk>]

Hi,

I'm finding my custom loader fails to deal with SUM() for fields returned as DataByteArray (as in PigStorage) when its LoadMetadata indicates the field type being float.

I've also changed all the field type to chararray which then doesn't find SUM function for chararray.

So is it true that automatic type conversion only kicked in when your loadFunc doesn't implement LoadMetadata?

i.e. if your loadFunc implements LoadMetadata, then the schema needs to tell the right type for the fields and the tuple it returns needs to hold the right type?

J

Re: Is automatic type conversion only kicked in when your loadFunc doesn't implement LoadMetadata?

[Daniel Dai <ji...@yahoo-inc.com>]

When you write a custom LoadFunc, you have two ways to determine the 
schema returned by LoadFunc:

1. Implemneting LoadMetadata, tell Pig the right schema your LoadFunc 
will produce. You need to convert LoadFunc output to match the schema 
you tell Pig

2. Not implementing LoadMetadata, Pig will assume that everything from 
your LoadFunc is bytearray. Pig will do the type conversion according to 
"AS" clause in your "Load" statement.

It seems you implement LoadMetadata, but in LoadFunc, you still return 
bytearray. If that's the case, either make conversion of data to match 
your declared schema, or not implementing LoadMetadata, let Pig do the 
type conversion.

Daniel

Jae Lee wrote:
> Hi,
>
> I'm finding my custom loader fails to deal with SUM() for fields returned as DataByteArray (as in PigStorage) when its LoadMetadata indicates the field type being float.
>
> I've also changed all the field type to chararray which then doesn't find SUM function for chararray.
>
> So is it true that automatic type conversion only kicked in when your loadFunc doesn't implement LoadMetadata?
>
> i.e. if your loadFunc implements LoadMetadata, then the schema needs to tell the right type for the fields and the tuple it returns needs to hold the right type?
>
> J
>
>