You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Vincent <vi...@gmail.com> on 2011/05/10 11:30:59 UTC
Tuple to lines conversion in Pig
Hello dear Pig users,
*I am loading a file with the following format:*
*$ cat peoples.txt
tom;1234,4567,6
anna;27894*
First field is a name, second field is a concatenation of an unknown number
of pets ids.
*I would like to JOIN this file with another one:*
*$ cat pets.txt
1234;dog;cocker
4567;mouse;usa
6;cat;persian
27894;cat;manx
*Fields are pet's id, pet's type, pet's race.
*
to get the following result:*
*1234;dog;cocker;tom
4567;mouse;usa;tom
6;cat;persian;tom
27894;cat;manx;anna*
*Problem is that I don't know how to convert a tuple of fields to lines,
i.e. to put the file peoples.txt into the following intermediate format:*
*tom,1234
tom,4567
tom,6
anna,27894*
Thanks in advance for your help!
Vincent Hervieux
Re: Tuple to lines conversion in Pig
Posted by Vincent <vi...@gmail.com>.
Thanks Yong and Mridul,
I was able to the the trick like this:
A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray, pets_ids
: chararray);
B = FOREACH A GENERATE name, TOKENIZE(REPLACE(pets_ids, ',', ' ')) AS
products_bag;
DUMP B;
DESCRIBE B;
C = FOREACH B GENERATE name, FLATTEN(products_bag) as (user_pet_id :
chararray);
DUMP C;
DESCRIBE C;
D = LOAD 'pets.txt' USING PigStorage(';') AS (id : chararray, type :
chararray, race: chararray);
DUMP D;
DESCRIBE D;
reqd_op = JOIN C BY user_pet_id, D BY id PARALLEL 5;
DUMP reqd_op;
For your info and maybe to help others, TOBAG(STRPLIT(...)) is creating a
bag containing only one tuple with all the fields, this is why it was not
working.
So you can do either:
*if you know the number of columns :*
B = FOREACH A GENERATE name, FLATTEN(STRPLIT(pets_ids, ',')) AS (id1 :
chararray, id2 : chararray, id3 : chararray);
C = FOREACH B GENERATE name, FLATTEN(TOBAG(id1,id2,id3));
DUMP C;
(tom,1234)
(tom,4567)
(tom,6)
(anna,27894)
(anna,)
(anna,)
*if you don't know the number of columns :*
B = FOREACH A GENERATE name, TOKENIZE(REPLACE(pets_ids, ',', ' ')) AS
products_bag;
C = FOREACH B GENERATE name, FLATTEN(products_bag) as (user_pet_id :
chararray);
(tom,1234)
(tom,4567)
(tom,6)
(anna,27894)
No null here.
I hope this can help some others.
Cheers
Vincent
On Tue, May 10, 2011 at 4:01 PM, Vincent <vi...@gmail.com> wrote:
> Hi Yong, Hi Mridul,
>
> I've changed everything to chararray:
>
>
> A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray, pets_ids
> : chararray);
>
> B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS pets_ids_separated;
> DUMP B;
> DESCRIBE B;
>
> C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS (id);
> DUMP C;
> DESCRIBE C;
>
> D = LOAD 'pets.txt' USING PigStorage(';') AS (id : chararray, type :
> chararray, race: chararray);
> DUMP D;
> DESCRIBE D;
>
> reqd_op = JOIN C BY id, D BY id PARALLEL 5;
> DUMP reqd_op;
>
> But I still have the error:
> 2011-05-10 15:59:42,472 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1107: Cannot merge join keys, incompatible types
> Details at logfile: /local/tmp/test/pig_expand/pig_1305028757759.log
>
> DUMP C is
> (tom,(1234,4567,6))
> (anna,(27894))
>
> Description is C: {name: chararray,id: (null)}
>
> How can JOIN operation can support on left hand a Tuple of chararray, and
> on right hand a chararray?
>
> BR
>
> Vincent
>
>
>
>
> On Tue, May 10, 2011 at 3:43 PM, 勇胡 <yo...@gmail.com> wrote:
>
>> You can see the type of join keys are different. One is chararray, the
>> other
>> is int. You have to change them into the same type.
>>
>> Yong
>>
>> 2011/5/10 Vincent <vi...@gmail.com>
>>
>> > According to your advices I wrote the following:
>> >
>> > *A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray,
>> > pets_ids
>> > : chararray);
>> >
>> > B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS
>> pets_ids_separated;
>> > DUMP B;
>> > DESCRIBE B;
>> >
>> > C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS
>> > user_pet_id;
>> > DUMP C;
>> > DESCRIBE C;
>> >
>> > D = LOAD 'pets.txt' USING PigStorage(';') AS (id : int, type :
>> chararray,
>> > race: chararray);
>> >
>> >
>> > reqd_op = JOIN C BY user_pet_id, D BY id PARALLEL 5;
>> > DUMP reqd_op;*
>> >
>> > But I have the following error:
>> > 2011-05-10 15:30:04,036 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>> > ERROR 1107: Cannot merge join keys, incompatible types
>> > Details at logfile: /local/tmp/test/pig_expand/pig_1305026987213.log
>> >
>> > Any idea, what it goes wrong here?
>> >
>> > Best Regards
>> >
>> > Vincent
>> >
>> >
>> >
>> > On Tue, May 10, 2011 at 3:04 PM, Mridul Muralidharan
>> > <mr...@yahoo-inc.com>wrote:
>> >
>> > >
>> > > I am not sure I follow your query related to PARALLEL.
>> > > The value for parallel is a static value.
>> > >
>> > > I was using $MY_PARALLEL as a placeholder to specify what sort of
>> > > parallelism you need.
>> > >
>> > > Typically you will have a default value in the script
>> > >
>> > > %default MY_PARALLEL '10'
>> > >
>> > > And override it, when required, using command line pig -param
>> > > MY_PARALLEL=50 ...
>> > >
>> > >
>> > >
>> > > Regards,
>> > > Mridul
>> > >
>> > >
>> > > On Tuesday 10 May 2011 04:26 PM, Vincent wrote:
>> > >
>> > >> Thanks Mridul for your quick answer!
>> > >>
>> > >> According to documentation PARALLEL is setting the number of reduce
>> > >> tasks. So how can I make it taking an UDF instead? Is there any
>> example
>> > >> of such functions in SVN/pig0.8 package?
>> > >>
>> > >> Best Regards
>> > >>
>> > >> Vincent
>> > >>
>> > >> On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
>> > >> <mridulm@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
>> > >>
>> > >>
>> > >> Easy option would be to write your own udf which can catch corner
>> > >> cases, etc ..
>> > >> But assuming your data strictly follows what you mentioned,
>> > >> something like this might help (illustrative only !) :
>> > >>
>> > >> pets = load 'pets.txt' USING PigStorage(';') AS
>> (pet_id:chararray,
>> > >> pet_type:chararray, pet_name:chararray);
>> > >>
>> > >> people = load 'peoples.txt' USING PigStorage(';') AS
>> > >> (user:chararray, ids:chararray);
>> > >> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
>> > >> -- STRSPLIT returns a tuple, not a bag : so convert to bag and
>> > >> flatten it.
>> > >> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1))
>> as
>> > >> (user_pet_id);
>> > >>
>> > >>
>> > >> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
>> > >> $MY_PARALLEL;
>> > >>
>> > >>
>> > >> reqd_op should contain what you need ...
>> > >>
>> > >>
>> > >>
>> > >> Regards,
>> > >> Mridul
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
>> > >>
>> > >> Hello dear Pig users,
>> > >>
>> > >> *I am loading a file with the following format:*
>> > >>
>> > >> *$ cat peoples.txt
>> > >> tom;1234,4567,6
>> > >> anna;27894*
>> > >> First field is a name, second field is a concatenation of an
>> > >> unknown number
>> > >> of pets ids.
>> > >>
>> > >> *I would like to JOIN this file with another one:*
>> > >>
>> > >> *$ cat pets.txt
>> > >> 1234;dog;cocker
>> > >> 4567;mouse;usa
>> > >> 6;cat;persian
>> > >> 27894;cat;manx
>> > >> *Fields are pet's id, pet's type, pet's race.
>> > >> *
>> > >> to get the following result:*
>> > >>
>> > >> *1234;dog;cocker;tom
>> > >> 4567;mouse;usa;tom
>> > >> 6;cat;persian;tom
>> > >> 27894;cat;manx;anna*
>> > >>
>> > >> *Problem is that I don't know how to convert a tuple of fields
>> > >> to lines,
>> > >> i.e. to put the file peoples.txt into the following
>> intermediate
>> > >> format:*
>> > >> *tom,1234
>> > >> tom,4567
>> > >> tom,6
>> > >> anna,27894*
>> > >>
>> > >> Thanks in advance for your help!
>> > >>
>> > >>
>> > >> Vincent Hervieux
>> > >>
>> > >>
>> > >>
>> > >>
>> > >
>> >
>>
>
>
Re: Tuple to lines conversion in Pig
Posted by Vincent <vi...@gmail.com>.
Hi Yong, Hi Mridul,
I've changed everything to chararray:
A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray, pets_ids
: chararray);
B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS pets_ids_separated;
DUMP B;
DESCRIBE B;
C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS (id);
DUMP C;
DESCRIBE C;
D = LOAD 'pets.txt' USING PigStorage(';') AS (id : chararray, type :
chararray, race: chararray);
DUMP D;
DESCRIBE D;
reqd_op = JOIN C BY id, D BY id PARALLEL 5;
DUMP reqd_op;
But I still have the error:
2011-05-10 15:59:42,472 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1107: Cannot merge join keys, incompatible types
Details at logfile: /local/tmp/test/pig_expand/pig_1305028757759.log
DUMP C is
(tom,(1234,4567,6))
(anna,(27894))
Description is C: {name: chararray,id: (null)}
How can JOIN operation can support on left hand a Tuple of chararray, and on
right hand a chararray?
BR
Vincent
On Tue, May 10, 2011 at 3:43 PM, 勇胡 <yo...@gmail.com> wrote:
> You can see the type of join keys are different. One is chararray, the
> other
> is int. You have to change them into the same type.
>
> Yong
>
> 2011/5/10 Vincent <vi...@gmail.com>
>
> > According to your advices I wrote the following:
> >
> > *A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray,
> > pets_ids
> > : chararray);
> >
> > B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS
> pets_ids_separated;
> > DUMP B;
> > DESCRIBE B;
> >
> > C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS
> > user_pet_id;
> > DUMP C;
> > DESCRIBE C;
> >
> > D = LOAD 'pets.txt' USING PigStorage(';') AS (id : int, type : chararray,
> > race: chararray);
> >
> >
> > reqd_op = JOIN C BY user_pet_id, D BY id PARALLEL 5;
> > DUMP reqd_op;*
> >
> > But I have the following error:
> > 2011-05-10 15:30:04,036 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 1107: Cannot merge join keys, incompatible types
> > Details at logfile: /local/tmp/test/pig_expand/pig_1305026987213.log
> >
> > Any idea, what it goes wrong here?
> >
> > Best Regards
> >
> > Vincent
> >
> >
> >
> > On Tue, May 10, 2011 at 3:04 PM, Mridul Muralidharan
> > <mr...@yahoo-inc.com>wrote:
> >
> > >
> > > I am not sure I follow your query related to PARALLEL.
> > > The value for parallel is a static value.
> > >
> > > I was using $MY_PARALLEL as a placeholder to specify what sort of
> > > parallelism you need.
> > >
> > > Typically you will have a default value in the script
> > >
> > > %default MY_PARALLEL '10'
> > >
> > > And override it, when required, using command line pig -param
> > > MY_PARALLEL=50 ...
> > >
> > >
> > >
> > > Regards,
> > > Mridul
> > >
> > >
> > > On Tuesday 10 May 2011 04:26 PM, Vincent wrote:
> > >
> > >> Thanks Mridul for your quick answer!
> > >>
> > >> According to documentation PARALLEL is setting the number of reduce
> > >> tasks. So how can I make it taking an UDF instead? Is there any
> example
> > >> of such functions in SVN/pig0.8 package?
> > >>
> > >> Best Regards
> > >>
> > >> Vincent
> > >>
> > >> On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
> > >> <mridulm@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
> > >>
> > >>
> > >> Easy option would be to write your own udf which can catch corner
> > >> cases, etc ..
> > >> But assuming your data strictly follows what you mentioned,
> > >> something like this might help (illustrative only !) :
> > >>
> > >> pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
> > >> pet_type:chararray, pet_name:chararray);
> > >>
> > >> people = load 'peoples.txt' USING PigStorage(';') AS
> > >> (user:chararray, ids:chararray);
> > >> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
> > >> -- STRSPLIT returns a tuple, not a bag : so convert to bag and
> > >> flatten it.
> > >> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
> > >> (user_pet_id);
> > >>
> > >>
> > >> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
> > >> $MY_PARALLEL;
> > >>
> > >>
> > >> reqd_op should contain what you need ...
> > >>
> > >>
> > >>
> > >> Regards,
> > >> Mridul
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
> > >>
> > >> Hello dear Pig users,
> > >>
> > >> *I am loading a file with the following format:*
> > >>
> > >> *$ cat peoples.txt
> > >> tom;1234,4567,6
> > >> anna;27894*
> > >> First field is a name, second field is a concatenation of an
> > >> unknown number
> > >> of pets ids.
> > >>
> > >> *I would like to JOIN this file with another one:*
> > >>
> > >> *$ cat pets.txt
> > >> 1234;dog;cocker
> > >> 4567;mouse;usa
> > >> 6;cat;persian
> > >> 27894;cat;manx
> > >> *Fields are pet's id, pet's type, pet's race.
> > >> *
> > >> to get the following result:*
> > >>
> > >> *1234;dog;cocker;tom
> > >> 4567;mouse;usa;tom
> > >> 6;cat;persian;tom
> > >> 27894;cat;manx;anna*
> > >>
> > >> *Problem is that I don't know how to convert a tuple of fields
> > >> to lines,
> > >> i.e. to put the file peoples.txt into the following
> intermediate
> > >> format:*
> > >> *tom,1234
> > >> tom,4567
> > >> tom,6
> > >> anna,27894*
> > >>
> > >> Thanks in advance for your help!
> > >>
> > >>
> > >> Vincent Hervieux
> > >>
> > >>
> > >>
> > >>
> > >
> >
>
Re: Tuple to lines conversion in Pig
Posted by 勇胡 <yo...@gmail.com>.
You can see the type of join keys are different. One is chararray, the other
is int. You have to change them into the same type.
Yong
2011/5/10 Vincent <vi...@gmail.com>
> According to your advices I wrote the following:
>
> *A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray,
> pets_ids
> : chararray);
>
> B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS pets_ids_separated;
> DUMP B;
> DESCRIBE B;
>
> C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS
> user_pet_id;
> DUMP C;
> DESCRIBE C;
>
> D = LOAD 'pets.txt' USING PigStorage(';') AS (id : int, type : chararray,
> race: chararray);
>
>
> reqd_op = JOIN C BY user_pet_id, D BY id PARALLEL 5;
> DUMP reqd_op;*
>
> But I have the following error:
> 2011-05-10 15:30:04,036 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1107: Cannot merge join keys, incompatible types
> Details at logfile: /local/tmp/test/pig_expand/pig_1305026987213.log
>
> Any idea, what it goes wrong here?
>
> Best Regards
>
> Vincent
>
>
>
> On Tue, May 10, 2011 at 3:04 PM, Mridul Muralidharan
> <mr...@yahoo-inc.com>wrote:
>
> >
> > I am not sure I follow your query related to PARALLEL.
> > The value for parallel is a static value.
> >
> > I was using $MY_PARALLEL as a placeholder to specify what sort of
> > parallelism you need.
> >
> > Typically you will have a default value in the script
> >
> > %default MY_PARALLEL '10'
> >
> > And override it, when required, using command line pig -param
> > MY_PARALLEL=50 ...
> >
> >
> >
> > Regards,
> > Mridul
> >
> >
> > On Tuesday 10 May 2011 04:26 PM, Vincent wrote:
> >
> >> Thanks Mridul for your quick answer!
> >>
> >> According to documentation PARALLEL is setting the number of reduce
> >> tasks. So how can I make it taking an UDF instead? Is there any example
> >> of such functions in SVN/pig0.8 package?
> >>
> >> Best Regards
> >>
> >> Vincent
> >>
> >> On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
> >> <mridulm@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
> >>
> >>
> >> Easy option would be to write your own udf which can catch corner
> >> cases, etc ..
> >> But assuming your data strictly follows what you mentioned,
> >> something like this might help (illustrative only !) :
> >>
> >> pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
> >> pet_type:chararray, pet_name:chararray);
> >>
> >> people = load 'peoples.txt' USING PigStorage(';') AS
> >> (user:chararray, ids:chararray);
> >> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
> >> -- STRSPLIT returns a tuple, not a bag : so convert to bag and
> >> flatten it.
> >> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
> >> (user_pet_id);
> >>
> >>
> >> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
> >> $MY_PARALLEL;
> >>
> >>
> >> reqd_op should contain what you need ...
> >>
> >>
> >>
> >> Regards,
> >> Mridul
> >>
> >>
> >>
> >>
> >>
> >> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
> >>
> >> Hello dear Pig users,
> >>
> >> *I am loading a file with the following format:*
> >>
> >> *$ cat peoples.txt
> >> tom;1234,4567,6
> >> anna;27894*
> >> First field is a name, second field is a concatenation of an
> >> unknown number
> >> of pets ids.
> >>
> >> *I would like to JOIN this file with another one:*
> >>
> >> *$ cat pets.txt
> >> 1234;dog;cocker
> >> 4567;mouse;usa
> >> 6;cat;persian
> >> 27894;cat;manx
> >> *Fields are pet's id, pet's type, pet's race.
> >> *
> >> to get the following result:*
> >>
> >> *1234;dog;cocker;tom
> >> 4567;mouse;usa;tom
> >> 6;cat;persian;tom
> >> 27894;cat;manx;anna*
> >>
> >> *Problem is that I don't know how to convert a tuple of fields
> >> to lines,
> >> i.e. to put the file peoples.txt into the following intermediate
> >> format:*
> >> *tom,1234
> >> tom,4567
> >> tom,6
> >> anna,27894*
> >>
> >> Thanks in advance for your help!
> >>
> >>
> >> Vincent Hervieux
> >>
> >>
> >>
> >>
> >
>
Re: Tuple to lines conversion in Pig
Posted by Vincent <vi...@gmail.com>.
According to your advices I wrote the following:
*A = LOAD 'peoples.txt' USING PigStorage(';') AS (name : chararray, pets_ids
: chararray);
B = foreach A GENERATE name, STRSPLIT(pets_ids, ',') AS pets_ids_separated;
DUMP B;
DESCRIBE B;
C = FOREACH B GENERATE name, FLATTEN(TOBAG(pets_ids_separated)) AS
user_pet_id;
DUMP C;
DESCRIBE C;
D = LOAD 'pets.txt' USING PigStorage(';') AS (id : int, type : chararray,
race: chararray);
reqd_op = JOIN C BY user_pet_id, D BY id PARALLEL 5;
DUMP reqd_op;*
But I have the following error:
2011-05-10 15:30:04,036 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1107: Cannot merge join keys, incompatible types
Details at logfile: /local/tmp/test/pig_expand/pig_1305026987213.log
Any idea, what it goes wrong here?
Best Regards
Vincent
On Tue, May 10, 2011 at 3:04 PM, Mridul Muralidharan
<mr...@yahoo-inc.com>wrote:
>
> I am not sure I follow your query related to PARALLEL.
> The value for parallel is a static value.
>
> I was using $MY_PARALLEL as a placeholder to specify what sort of
> parallelism you need.
>
> Typically you will have a default value in the script
>
> %default MY_PARALLEL '10'
>
> And override it, when required, using command line pig -param
> MY_PARALLEL=50 ...
>
>
>
> Regards,
> Mridul
>
>
> On Tuesday 10 May 2011 04:26 PM, Vincent wrote:
>
>> Thanks Mridul for your quick answer!
>>
>> According to documentation PARALLEL is setting the number of reduce
>> tasks. So how can I make it taking an UDF instead? Is there any example
>> of such functions in SVN/pig0.8 package?
>>
>> Best Regards
>>
>> Vincent
>>
>> On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
>> <mridulm@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
>>
>>
>> Easy option would be to write your own udf which can catch corner
>> cases, etc ..
>> But assuming your data strictly follows what you mentioned,
>> something like this might help (illustrative only !) :
>>
>> pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
>> pet_type:chararray, pet_name:chararray);
>>
>> people = load 'peoples.txt' USING PigStorage(';') AS
>> (user:chararray, ids:chararray);
>> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
>> -- STRSPLIT returns a tuple, not a bag : so convert to bag and
>> flatten it.
>> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
>> (user_pet_id);
>>
>>
>> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
>> $MY_PARALLEL;
>>
>>
>> reqd_op should contain what you need ...
>>
>>
>>
>> Regards,
>> Mridul
>>
>>
>>
>>
>>
>> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
>>
>> Hello dear Pig users,
>>
>> *I am loading a file with the following format:*
>>
>> *$ cat peoples.txt
>> tom;1234,4567,6
>> anna;27894*
>> First field is a name, second field is a concatenation of an
>> unknown number
>> of pets ids.
>>
>> *I would like to JOIN this file with another one:*
>>
>> *$ cat pets.txt
>> 1234;dog;cocker
>> 4567;mouse;usa
>> 6;cat;persian
>> 27894;cat;manx
>> *Fields are pet's id, pet's type, pet's race.
>> *
>> to get the following result:*
>>
>> *1234;dog;cocker;tom
>> 4567;mouse;usa;tom
>> 6;cat;persian;tom
>> 27894;cat;manx;anna*
>>
>> *Problem is that I don't know how to convert a tuple of fields
>> to lines,
>> i.e. to put the file peoples.txt into the following intermediate
>> format:*
>> *tom,1234
>> tom,4567
>> tom,6
>> anna,27894*
>>
>> Thanks in advance for your help!
>>
>>
>> Vincent Hervieux
>>
>>
>>
>>
>
Re: Tuple to lines conversion in Pig
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
I am not sure I follow your query related to PARALLEL.
The value for parallel is a static value.
I was using $MY_PARALLEL as a placeholder to specify what sort of
parallelism you need.
Typically you will have a default value in the script
%default MY_PARALLEL '10'
And override it, when required, using command line pig -param
MY_PARALLEL=50 ...
Regards,
Mridul
On Tuesday 10 May 2011 04:26 PM, Vincent wrote:
> Thanks Mridul for your quick answer!
>
> According to documentation PARALLEL is setting the number of reduce
> tasks. So how can I make it taking an UDF instead? Is there any example
> of such functions in SVN/pig0.8 package?
>
> Best Regards
>
> Vincent
>
> On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
> <mridulm@yahoo-inc.com <ma...@yahoo-inc.com>> wrote:
>
>
> Easy option would be to write your own udf which can catch corner
> cases, etc ..
> But assuming your data strictly follows what you mentioned,
> something like this might help (illustrative only !) :
>
> pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
> pet_type:chararray, pet_name:chararray);
>
> people = load 'peoples.txt' USING PigStorage(';') AS
> (user:chararray, ids:chararray);
> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
> -- STRSPLIT returns a tuple, not a bag : so convert to bag and
> flatten it.
> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
> (user_pet_id);
>
>
> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
> $MY_PARALLEL;
>
>
> reqd_op should contain what you need ...
>
>
>
> Regards,
> Mridul
>
>
>
>
>
> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
>
> Hello dear Pig users,
>
> *I am loading a file with the following format:*
>
> *$ cat peoples.txt
> tom;1234,4567,6
> anna;27894*
> First field is a name, second field is a concatenation of an
> unknown number
> of pets ids.
>
> *I would like to JOIN this file with another one:*
>
> *$ cat pets.txt
> 1234;dog;cocker
> 4567;mouse;usa
> 6;cat;persian
> 27894;cat;manx
> *Fields are pet's id, pet's type, pet's race.
> *
> to get the following result:*
>
> *1234;dog;cocker;tom
> 4567;mouse;usa;tom
> 6;cat;persian;tom
> 27894;cat;manx;anna*
>
> *Problem is that I don't know how to convert a tuple of fields
> to lines,
> i.e. to put the file peoples.txt into the following intermediate
> format:*
> *tom,1234
> tom,4567
> tom,6
> anna,27894*
>
> Thanks in advance for your help!
>
>
> Vincent Hervieux
>
>
>
Re: Tuple to lines conversion in Pig
Posted by Vincent <vi...@gmail.com>.
Thanks Mridul for your quick answer!
According to documentation PARALLEL is setting the number of reduce tasks.
So how can I make it taking an UDF instead? Is there any example of such
functions in SVN/pig0.8 package?
Best Regards
Vincent
On Tue, May 10, 2011 at 2:02 PM, Mridul Muralidharan
<mr...@yahoo-inc.com>wrote:
>
> Easy option would be to write your own udf which can catch corner cases,
> etc ..
> But assuming your data strictly follows what you mentioned, something like
> this might help (illustrative only !) :
>
> pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
> pet_type:chararray, pet_name:chararray);
>
> people = load 'peoples.txt' USING PigStorage(';') AS (user:chararray,
> ids:chararray);
> people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
> -- STRSPLIT returns a tuple, not a bag : so convert to bag and flatten it.
> people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
> (user_pet_id);
>
>
> reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
> $MY_PARALLEL;
>
>
> reqd_op should contain what you need ...
>
>
>
> Regards,
> Mridul
>
>
>
>
>
> On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
>
>> Hello dear Pig users,
>>
>> *I am loading a file with the following format:*
>>
>> *$ cat peoples.txt
>> tom;1234,4567,6
>> anna;27894*
>> First field is a name, second field is a concatenation of an unknown
>> number
>> of pets ids.
>>
>> *I would like to JOIN this file with another one:*
>>
>> *$ cat pets.txt
>> 1234;dog;cocker
>> 4567;mouse;usa
>> 6;cat;persian
>> 27894;cat;manx
>> *Fields are pet's id, pet's type, pet's race.
>> *
>> to get the following result:*
>>
>> *1234;dog;cocker;tom
>> 4567;mouse;usa;tom
>> 6;cat;persian;tom
>> 27894;cat;manx;anna*
>>
>> *Problem is that I don't know how to convert a tuple of fields to lines,
>> i.e. to put the file peoples.txt into the following intermediate format:*
>> *tom,1234
>> tom,4567
>> tom,6
>> anna,27894*
>>
>> Thanks in advance for your help!
>>
>>
>> Vincent Hervieux
>>
>
>
Re: Tuple to lines conversion in Pig
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
Easy option would be to write your own udf which can catch corner cases,
etc ..
But assuming your data strictly follows what you mentioned, something
like this might help (illustrative only !) :
pets = load 'pets.txt' USING PigStorage(';') AS (pet_id:chararray,
pet_type:chararray, pet_name:chararray);
people = load 'peoples.txt' USING PigStorage(';') AS (user:chararray,
ids:chararray);
people_t = FOREACH people GENERATE user, STRSPLIT(ids, ',');
-- STRSPLIT returns a tuple, not a bag : so convert to bag and flatten it.
people_reqd = FOREACH people_t GENERATE user, FLATTEN(TOBAG($1)) as
(user_pet_id);
reqd_op = JOIN people_reqd BY user_pet_id, pets BY pet_id PARALLEL
$MY_PARALLEL;
reqd_op should contain what you need ...
Regards,
Mridul
On Tuesday 10 May 2011 03:00 PM, Vincent wrote:
> Hello dear Pig users,
>
> *I am loading a file with the following format:*
>
> *$ cat peoples.txt
> tom;1234,4567,6
> anna;27894*
> First field is a name, second field is a concatenation of an unknown number
> of pets ids.
>
> *I would like to JOIN this file with another one:*
>
> *$ cat pets.txt
> 1234;dog;cocker
> 4567;mouse;usa
> 6;cat;persian
> 27894;cat;manx
> *Fields are pet's id, pet's type, pet's race.
> *
> to get the following result:*
>
> *1234;dog;cocker;tom
> 4567;mouse;usa;tom
> 6;cat;persian;tom
> 27894;cat;manx;anna*
>
> *Problem is that I don't know how to convert a tuple of fields to lines,
> i.e. to put the file peoples.txt into the following intermediate format:*
> *tom,1234
> tom,4567
> tom,6
> anna,27894*
>
> Thanks in advance for your help!
>
>
> Vincent Hervieux