You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Corbin Hoenes <co...@tynt.com> on 2010/07/23 19:15:42 UTC
best way for pig and mapreduce jobs to be used interchangeably
What are some strategies to have pig and java mapreduce jobs exchange data? E.g. we find a particular pig script in a chain is too slow and we could optimize with a custom mapreduce job we'd want pig to write the data out in a format that mapreduce could access and vice versa.
Re: best way for pig and mapreduce jobs to be used interchangeably
Posted by Ashutosh Chauhan <as...@gmail.com>.
On Wed, Jul 28, 2010 at 10:37, Scott Carey <sc...@richrelevance.com> wrote:
> In the long term, I'd like to be able to share such data between Java M/R, Hive, Pig, and Cascading relatively easily and not rely on > text formats, which have issues with delimiters/escapes, performance, and schema maintenance.
This is one of the major reason we are building Howl
http://wiki.apache.org/pig/Howl
Mailing list: http://tech.groups.yahoo.com/group/howldev/
Ashutosh
Re: best way for pig and mapreduce jobs to be used interchangeably
Posted by Scott Carey <sc...@richrelevance.com>.
Currently we use Text (in Pig 0.7+, PigStorage just wraps Hadoop's text output format). But this can be annoying to maintain. Column changes or reordering break things, and the output itself does not contain information about what the columns are. Some of our intermediates last for a year or more, and having to re-generate them when enhancements are made is annoying.
I'm in the process of writing an Avro reader/writer for Pig (https://issues.apache.org/jira/browse/AVRO-592), but it most likely won't make it into Avro 1.4 -- I don't have time to work on it for a few weeks.
This would make it easier to maintain intermediate data since the schema would be persisted in the files -- column ordering and schema evolution is a part of Avro so your Java and Pig don't have to synchronize, and 'long term' intermediate outputs don't necessarily have to be re-generated when the code changes. In the long term, I'd like to be able to share such data between Java M/R, Hive, Pig, and Cascading relatively easily and not rely on text formats, which have issues with delimiters/escapes, performance, and schema maintenance.
On Jul 28, 2010, at 8:25 AM, Corbin Hoenes wrote:
> Mridul -
>
> What file format do you use to exchange data between pig and java? Text or something else?
>
> On Jul 25, 2010, at 1:52 PM, Mridul Muralidharan wrote:
>
>>
>>
>> In some of our pipelines, pig jobs are part of the pipeline - which consist of other hadoop jobs, shell executions, etc.
>> We currently do this by using intermediate file dumps.
>>
>>
>> Regards,
>> Mridul
>>
>>
>>
>> On Friday 23 July 2010 10:45 PM, Corbin Hoenes wrote:
>>> What are some strategies to have pig and java mapreduce jobs exchange data? E.g. we find a particular pig script in a chain is too slow and we could optimize with a custom mapreduce job we'd want pig to write the data out in a format that mapreduce could access and vice versa.
>>>
>>
>
Re: best way for pig and mapreduce jobs to be used interchangeably
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
I think we have cases where it is a writable (sent as bytearray to pig
and stored as sequence files iirc - hardcoded schema for the output),
and plain csv text output.
Text output has issues due to it being not very 'nice' in terms of
schema evolution, but extremely simple to code it up for prototyping !
The only constraint is to have appropriate LoadFunc/StoreFunc [1] for
pig and appropriate input format/reader for hadoop.
Regards,
Mridul
[1] LoadFunc is not really required, but helps in case you want to try
other pig jobs on the intermediate output.
On Wednesday 28 July 2010 08:55 PM, Corbin Hoenes wrote:
> Mridul -
>
> What file format do you use to exchange data between pig and java? Text or something else?
>
> On Jul 25, 2010, at 1:52 PM, Mridul Muralidharan wrote:
>
>>
>>
>> In some of our pipelines, pig jobs are part of the pipeline - which consist of other hadoop jobs, shell executions, etc.
>> We currently do this by using intermediate file dumps.
>>
>>
>> Regards,
>> Mridul
>>
>>
>>
>> On Friday 23 July 2010 10:45 PM, Corbin Hoenes wrote:
>>> What are some strategies to have pig and java mapreduce jobs exchange data? E.g. we find a particular pig script in a chain is too slow and we could optimize with a custom mapreduce job we'd want pig to write the data out in a format that mapreduce could access and vice versa.
>>>
>>
>
Re: best way for pig and mapreduce jobs to be used interchangeably
Posted by Corbin Hoenes <co...@tynt.com>.
Mridul -
What file format do you use to exchange data between pig and java? Text or something else?
On Jul 25, 2010, at 1:52 PM, Mridul Muralidharan wrote:
>
>
> In some of our pipelines, pig jobs are part of the pipeline - which consist of other hadoop jobs, shell executions, etc.
> We currently do this by using intermediate file dumps.
>
>
> Regards,
> Mridul
>
>
>
> On Friday 23 July 2010 10:45 PM, Corbin Hoenes wrote:
>> What are some strategies to have pig and java mapreduce jobs exchange data? E.g. we find a particular pig script in a chain is too slow and we could optimize with a custom mapreduce job we'd want pig to write the data out in a format that mapreduce could access and vice versa.
>>
>
Re: best way for pig and mapreduce jobs to be used interchangeably
Posted by Mridul Muralidharan <mr...@yahoo-inc.com>.
In some of our pipelines, pig jobs are part of the pipeline - which
consist of other hadoop jobs, shell executions, etc.
We currently do this by using intermediate file dumps.
Regards,
Mridul
On Friday 23 July 2010 10:45 PM, Corbin Hoenes wrote:
> What are some strategies to have pig and java mapreduce jobs exchange data? E.g. we find a particular pig script in a chain is too slow and we could optimize with a custom mapreduce job we'd want pig to write the data out in a format that mapreduce could access and vice versa.
>