You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dexin Wang <wa...@gmail.com> on 2010/12/21 22:23:25 UTC
set reducer timeout with pig
Hi,
How do I change the default timeout for reducer with Pig? I have some
reducer that needs to take longer than 10 minutes to finish. It is pretty
frustrating to see many of get to 95% complete and then got killed.
Thanks.
Dexin
Re: set reducer timeout with pig
Posted by Ashutosh Chauhan <ha...@apache.org>.
SET is only available in 0.8 not in 0.7. But I think your problem lies
elsewhere and most likely is not a reporting timeout issue.
Ashutosh
On Wednesday, December 22, 2010, Dexin Wang <wa...@gmail.com> wrote:
> It doesn't seem to work. I got
>
> 2010-12-22 21:36:59,120 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1000: Error during parsing. Unrecognized set key: mapred.task.timeout
>
> I did this in my pig script:
>
> SET mapred.task.timeout 1800000;
>
> What did I do wrong? Thanks.
>
> % pig --version
> Apache Pig version 0.7.0+9 (rexported)
> compiled Jun 28 2010, 12:53:50
>
> On Tue, Dec 21, 2010 at 2:39 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
>
>> True, however there is one bug in 0.7. We fix it in 0.8.
>>
>> https://issues.apache.org/jira/browse/PIG-1760
>>
>> Daniel
>>
>>
>>
>> Ashutosh Chauhan wrote:
>>
>>> Ideally you need not to do that. Pig automatically takes care of
>>> progress reporting in its operator. Do you have a pig script which
>>> fails because of reporting progress timeout issues ?
>>>
>>> Ashutosh
>>>
>>> On Tue, Dec 21, 2010 at 13:23, Dexin Wang <wa...@gmail.com> wrote:
>>>
>>>
>>>> Hi,
>>>>
>>>> How do I change the default timeout for reducer with Pig? I have some
>>>> reducer that needs to take longer than 10 minutes to finish. It is pretty
>>>> frustrating to see many of get to 95% complete and then got killed.
>>>>
>>>> Thanks.
>>>> Dexin
>>>>
>>>>
>>>>
>>>
>>
>
Re: set reducer timeout with pig
Posted by Dexin Wang <wa...@gmail.com>.
It doesn't seem to work. I got
2010-12-22 21:36:59,120 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1000: Error during parsing. Unrecognized set key: mapred.task.timeout
I did this in my pig script:
SET mapred.task.timeout 1800000;
What did I do wrong? Thanks.
% pig --version
Apache Pig version 0.7.0+9 (rexported)
compiled Jun 28 2010, 12:53:50
On Tue, Dec 21, 2010 at 2:39 PM, Daniel Dai <ji...@yahoo-inc.com> wrote:
> True, however there is one bug in 0.7. We fix it in 0.8.
>
> https://issues.apache.org/jira/browse/PIG-1760
>
> Daniel
>
>
>
> Ashutosh Chauhan wrote:
>
>> Ideally you need not to do that. Pig automatically takes care of
>> progress reporting in its operator. Do you have a pig script which
>> fails because of reporting progress timeout issues ?
>>
>> Ashutosh
>>
>> On Tue, Dec 21, 2010 at 13:23, Dexin Wang <wa...@gmail.com> wrote:
>>
>>
>>> Hi,
>>>
>>> How do I change the default timeout for reducer with Pig? I have some
>>> reducer that needs to take longer than 10 minutes to finish. It is pretty
>>> frustrating to see many of get to 95% complete and then got killed.
>>>
>>> Thanks.
>>> Dexin
>>>
>>>
>>>
>>
>
Re: set reducer timeout with pig
Posted by Daniel Dai <ji...@yahoo-inc.com>.
True, however there is one bug in 0.7. We fix it in 0.8.
https://issues.apache.org/jira/browse/PIG-1760
Daniel
Ashutosh Chauhan wrote:
> Ideally you need not to do that. Pig automatically takes care of
> progress reporting in its operator. Do you have a pig script which
> fails because of reporting progress timeout issues ?
>
> Ashutosh
>
> On Tue, Dec 21, 2010 at 13:23, Dexin Wang <wa...@gmail.com> wrote:
>
>> Hi,
>>
>> How do I change the default timeout for reducer with Pig? I have some
>> reducer that needs to take longer than 10 minutes to finish. It is pretty
>> frustrating to see many of get to 95% complete and then got killed.
>>
>> Thanks.
>> Dexin
>>
>>
Re: set reducer timeout with pig
Posted by Ashutosh Chauhan <ha...@apache.org>.
Ideally you need not to do that. Pig automatically takes care of
progress reporting in its operator. Do you have a pig script which
fails because of reporting progress timeout issues ?
Ashutosh
On Tue, Dec 21, 2010 at 13:23, Dexin Wang <wa...@gmail.com> wrote:
> Hi,
>
> How do I change the default timeout for reducer with Pig? I have some
> reducer that needs to take longer than 10 minutes to finish. It is pretty
> frustrating to see many of get to 95% complete and then got killed.
>
> Thanks.
> Dexin
>
Re: set reducer timeout with pig
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
"set mapred.task.timeout $num_milliseconds"
On Tue, Dec 21, 2010 at 1:23 PM, Dexin Wang <wa...@gmail.com> wrote:
> Hi,
>
> How do I change the default timeout for reducer with Pig? I have some
> reducer that needs to take longer than 10 minutes to finish. It is pretty
> frustrating to see many of get to 95% complete and then got killed.
>
> Thanks.
> Dexin
>