You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@pig.apache.org by abhishek dodda <ab...@gmail.com> on 2013/06/16 04:44:11 UTC

pig union with avro

hello,

I am doing this

DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();

A = load '/user/abhi/a.txt' using AVRO_LOAD;

B = load '/user/abhi/b.txt' using AVRO_LOAD;

C = UNION A , B;

here script is failing with the following error

ERROR org.apache.pig.tools.grunt.Grunt - ERROR

Re: pig union with avro

Posted by abhishek <ab...@gmail.com>.

Thanks for reply Cheolsoo 

What's the output of describe A and B?

Output of A and B are not identical. 

If the schema of A and B are not identical, union will type-cast fields to merge them

http://pig.apache.org/docs/r0.11.1/basic.html#union

> I would try to find which fields pig tries to cast to bytearray while
> merging them. Casting anything to bytearray is invalid resulting this error:

I don't exactly understand the above point.

Is it some thing related to PIG 2493

https://issues.apache.org/jira/browse/PIG-2493

This is resolved in pig 0.10 ??

cannot cast to byte array

Thanks,
Cheolsoo
> 
> On Sat, Jun 15, 2013 at 7:48 PM, abhishek dodda
> <ab...@gmail.com>wrote:
> 
>> hello,
>> 
>> I am doing this
>> 
>> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>> 
>> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>> 
>> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>> 
>> C = UNION A , B;
>> 
>> here script is failing with the following error
>> 
>> *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1051 : cannot cast to byte
>> array*
>> 
>> In the pig logs the error is
>> 
>> *ERROR 1056 problem while casting inputs of union*.
>> 
>> Script was running fine before, but it is failing now with the above error
>> 
>> Regards
>> abhishek
>> 
>> 
>> On Sat, Jun 15, 2013 at 7:44 PM, abhishek dodda
>> <ab...@gmail.com>wrote:
>> 
>>> hello,
>>> 
>>> I am doing this
>>> 
>>> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>>> 
>>> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>>> 
>>> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>>> 
>>> C = UNION A , B;
>>> 
>>> here script is failing with the following error
>>> 
>>> ERROR org.apache.pig.tools.grunt.Grunt - ERROR
>>

Re: pig union with avro

Posted by abhishek <ab...@gmail.com>.

Cheolsoo,

Thank you very much , I found the in compatible types in the union.

Now it is working fine.


Regards
Abhishek

Sent from my iPhone

On Jun 15, 2013, at 11:52 PM, Cheolsoo Park <pi...@gmail.com> wrote:

> What's the output of describe A and B?
> 
> If the schema of A and B are not identical, union will type-cast fields to
> merge them:
> 
> http://pig.apache.org/docs/r0.11.1/basic.html#union
> 
> I would try to find which fields Pig tries to cast to bytearray while
> merging them. Casting anything to bytearray is invalid resulting this error:
> 
>>> cannot cast to byte array
> 
> Thanks,
> Cheolsoo
> 
> 
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 7:48 PM, abhishek dodda
> <ab...@gmail.com>wrote:
> 
>> hello,
>> 
>> I am doing this
>> 
>> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>> 
>> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>> 
>> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>> 
>> C = UNION A , B;
>> 
>> here script is failing with the following error
>> 
>> *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1051 : cannot cast to byte
>> array*
>> 
>> In the pig logs the error is
>> 
>> *ERROR 1056 problem while casting inputs of union*.
>> 
>> Script was running fine before, but it is failing now with the above error
>> 
>> Regards
>> abhishek
>> 
>> 
>> On Sat, Jun 15, 2013 at 7:44 PM, abhishek dodda
>> <ab...@gmail.com>wrote:
>> 
>>> hello,
>>> 
>>> I am doing this
>>> 
>>> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>>> 
>>> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>>> 
>>> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>>> 
>>> C = UNION A , B;
>>> 
>>> here script is failing with the following error
>>> 
>>> ERROR org.apache.pig.tools.grunt.Grunt - ERROR
>>

Re: pig union with avro

Posted by Cheolsoo Park <pi...@gmail.com>.

What's the output of describe A and B?

If the schema of A and B are not identical, union will type-cast fields to
merge them:

http://pig.apache.org/docs/r0.11.1/basic.html#union

I would try to find which fields Pig tries to cast to bytearray while
merging them. Casting anything to bytearray is invalid resulting this error:

>> cannot cast to byte array

Thanks,
Cheolsoo






On Sat, Jun 15, 2013 at 7:48 PM, abhishek dodda
<ab...@gmail.com>wrote:

> hello,
>
> I am doing this
>
> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>
> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>
> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>
> C = UNION A , B;
>
> here script is failing with the following error
>
> *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1051 : cannot cast to byte
> array*
>
> In the pig logs the error is
>
> *ERROR 1056 problem while casting inputs of union*.
>
> Script was running fine before, but it is failing now with the above error
>
> Regards
> abhishek
>
>
> On Sat, Jun 15, 2013 at 7:44 PM, abhishek dodda
> <ab...@gmail.com>wrote:
>
> > hello,
> >
> > I am doing this
> >
> > DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
> >
> > A = load '/user/abhi/a.txt' using AVRO_LOAD;
> >
> > B = load '/user/abhi/b.txt' using AVRO_LOAD;
> >
> > C = UNION A , B;
> >
> > here script is failing with the following error
> >
> > ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> >
>

Re: pig union with avro

Posted by abhishek dodda <ab...@gmail.com>.

hello,

I am doing this

DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();

A = load '/user/abhi/a.txt' using AVRO_LOAD;

B = load '/user/abhi/b.txt' using AVRO_LOAD;

C = UNION A , B;

here script is failing with the following error

*ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1051 : cannot cast to byte
array*

In the pig logs the error is

*ERROR 1056 problem while casting inputs of union*.

Script was running fine before, but it is failing now with the above error

Regards
abhishek


On Sat, Jun 15, 2013 at 7:44 PM, abhishek dodda
<ab...@gmail.com>wrote:

> hello,
>
> I am doing this
>
> DEFINE AVRO_LOAD org.apache.pig.piggybank.strorage.avro.AvroStorage();
>
> A = load '/user/abhi/a.txt' using AVRO_LOAD;
>
> B = load '/user/abhi/b.txt' using AVRO_LOAD;
>
> C = UNION A , B;
>
> here script is failing with the following error
>
> ERROR org.apache.pig.tools.grunt.Grunt - ERROR
>