You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Haitao Yao <ya...@gmail.com> on 2012/02/20 07:58:58 UTC
How to run pig batch?
hi, all
I have tens of simple pig scripts to run. While there's no parameter name collision, I merged them into a large pig script which is about 4000 lines.
But the merged pig script takes pig a lot of time merging and analyzing.
How can I quickly submit tens of scripts in parallel?
BTW, I do not want to start tens of pig process for memory issue.
thanks very much.
Re: How to run pig batch?
Posted by Haitao Yao <ya...@gmail.com>.
the pig script is generated by a system which is more than 4000 lines.
The pig client compile it for a long time.
I have solved the problem by dividing the big script into small ones and implementing a new pig client which will concurrently submit multiple pig scripts for computation.
Is there any other way?
thanks.
在 2012-2-21，上午5:02， Dmitriy Ryaboy 写道：
> Is something preventing you from calling them via a simple bash script?
>
> On Sun, Feb 19, 2012 at 10:58 PM, Haitao Yao <ya...@gmail.com> wrote:
>> hi, all
>> I have tens of simple pig scripts to run. While there's no parameter name collision, I merged them into a large pig script which is about 4000 lines.
>> But the merged pig script takes pig a lot of time merging and analyzing.
>>
>> How can I quickly submit tens of scripts in parallel?
>>
>> BTW, I do not want to start tens of pig process for memory issue.
>> thanks very much.
Re: How to run pig batch?
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Is something preventing you from calling them via a simple bash script?
On Sun, Feb 19, 2012 at 10:58 PM, Haitao Yao <ya...@gmail.com> wrote:
> hi, all
> I have tens of simple pig scripts to run. While there's no parameter name collision, I merged them into a large pig script which is about 4000 lines.
> But the merged pig script takes pig a lot of time merging and analyzing.
>
> How can I quickly submit tens of scripts in parallel?
>
> BTW, I do not want to start tens of pig process for memory issue.
> thanks very much.