Does pig stable enough in local model

[zhang jianfeng <zj...@gmail.com>]

Hi all,

I meet a problem in local model, when using a small data, it works fine, but
when I run larger data, the pig will hang there,

And it's difficult for me to analysis why it hangs and where it hangs, does
anyone know this problem, and does pig insure the local model, or it is only
a development model, not applicable for product ?


Thank you .

Re: Does pig stable enough in local model

[Mridul Muralidharan <mr...@yahoo-inc.com>]

Hi,

   IIRC pig tries to evaluate the entire script in one go when in local 
mode ... as opposed to splitting it at mapred boundaries.
This could be faster in some cases, but in others - where there are lot 
of intermediate tuples (imo) - it will pretty much slow it to a crawl 
when compared to mapred case : particularly when the bags start spilling 
to disk.

Are you observing databag spills to disk ? Or is your script really deep ?
Though it could also be the bug Alan mentioned !

Regards,
Mridul


zhang jianfeng wrote:
> Hi all,
> 
> I meet a problem in local model, when using a small data, it works fine, but
> when I run larger data, the pig will hang there,
> 
> And it's difficult for me to analysis why it hangs and where it hangs, does
> anyone know this problem, and does pig insure the local model, or it is only
> a development model, not applicable for product ?
> 
> 
> Thank you .
> 

Re: Does pig stable enough in local model

[zhang jianfeng <zj...@gmail.com>]

Alan,

Thank you for your help, It works after I update the pig source code,



On Mon, May 11, 2009 at 1:07 PM, Alan Gates <ga...@yahoo-inc.com> wrote:

> How high are you setting the maximum memory for your JVM when you run it
> with larger data?  Also, are there any distinct or order by in your program?
>  There's a known bug (https://issues.apache.org/jira/browse/PIG-800) that
> causes infinite loops in local mode in some cases.
>
> Alan.
>
>
> On May 10, 2009, at 7:59 PM, zhang jianfeng wrote:
>
>  Hi all,
>>
>> I meet a problem in local model, when using a small data, it works fine,
>> but
>> when I run larger data, the pig will hang there,
>>
>> And it's difficult for me to analysis why it hangs and where it hangs,
>> does
>> anyone know this problem, and does pig insure the local model, or it is
>> only
>> a development model, not applicable for product ?
>>
>>
>> Thank you .
>>
>
>

Re: Does pig stable enough in local model

[Alan Gates <ga...@yahoo-inc.com>]

How high are you setting the maximum memory for your JVM when you run  
it with larger data?  Also, are there any distinct or order by in your  
program?  There's a known bug (https://issues.apache.org/jira/browse/PIG-800 
) that causes infinite loops in local mode in some cases.

Alan.

On May 10, 2009, at 7:59 PM, zhang jianfeng wrote:

> Hi all,
>
> I meet a problem in local model, when using a small data, it works  
> fine, but
> when I run larger data, the pig will hang there,
>
> And it's difficult for me to analysis why it hangs and where it  
> hangs, does
> anyone know this problem, and does pig insure the local model, or it  
> is only
> a development model, not applicable for product ?
>
>
> Thank you .