You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by canan chen <cc...@gmail.com> on 2015/01/16 12:28:01 UTC
Pig on mapreduce depends on MR historyserver ?
I try to run pig on mapreduce, but would meet the following error if I
didn't start mr history server.
But I can run hive on mapreduce without MR historyserver, not sure why pig
on mapreduce on MR historyserver. As my understanding historyserver is used
for recovery, so it should be OK without historyserver, right ?
2015-01-16 18:38:48,642 [main] INFO
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline
service address: http://0.0.0.0:8188/ws/v1/timeline/
2015-01-16 18:38:48,643 [main] INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at /0.0.0.0:8032
2015-01-16 18:38:48,650 [main] INFO
org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
2015-01-16 18:38:49,653 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/0.0.0.0:10020. Already tried 0
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2015-01-16 18:38:50,656 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/0.0.0.0:10020. Already tried 1
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2015-01-16 18:38:51,658 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/0.0.0.0:10020. Already tried 2
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
2015-01-16 18:38:52,661 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/0.0.0.0:10020. Already tried 3
time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1000 MILLISECONDS)
Re: Pig on mapreduce depends on MR historyserver ?
Posted by Daniel Dai <da...@hortonworks.com>.
Pig does not depends on MR historyserver directly, but MR framework does.
Pig needs to pull counter/diagnose information for each MR job it submits
from AM, but if AM dies, MR client will redirect the request to history
server. Note in Pig 0.13 and prior, there is a bug which keep Pig trying
history server for a long time
(http://issues.apache.org/jira/browse/PIG-4164).
Daniel
On 1/27/15, 6:07 PM, "canan chen" <cc...@gmail.com> wrote:
>Has anyone experience this ? Thanks
>
>On Fri, Jan 16, 2015 at 7:28 PM, canan chen <cc...@gmail.com> wrote:
>
>> I try to run pig on mapreduce, but would meet the following error if I
>> didn't start mr history server.
>> But I can run hive on mapreduce without MR historyserver, not sure why
>>pig
>> on mapreduce on MR historyserver. As my understanding historyserver is
>>used
>> for recovery, so it should be OK without historyserver, right ?
>>
>>
>> 2015-01-16 18:38:48,642 [main] INFO
>> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline
>> service address: http://0.0.0.0:8188/ws/v1/timeline/
>> 2015-01-16 18:38:48,643 [main] INFO
>>org.apache.hadoop.yarn.client.RMProxy
>> - Connecting to ResourceManager at /0.0.0.0:8032
>> 2015-01-16 18:38:48,650 [main] INFO
>> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
>> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
>> server
>> 2015-01-16 18:38:49,653 [main] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 0
>> time(s); retry policy is
>>RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)
>> 2015-01-16 18:38:50,656 [main] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 1
>> time(s); retry policy is
>>RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)
>> 2015-01-16 18:38:51,658 [main] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 2
>> time(s); retry policy is
>>RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)
>> 2015-01-16 18:38:52,661 [main] INFO org.apache.hadoop.ipc.Client -
>> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 3
>> time(s); retry policy is
>>RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>> sleepTime=1000 MILLISECONDS)
>>
>>
>>
>>
Re: Pig on mapreduce depends on MR historyserver ?
Posted by canan chen <cc...@gmail.com>.
Has anyone experience this ? Thanks
On Fri, Jan 16, 2015 at 7:28 PM, canan chen <cc...@gmail.com> wrote:
> I try to run pig on mapreduce, but would meet the following error if I
> didn't start mr history server.
> But I can run hive on mapreduce without MR historyserver, not sure why pig
> on mapreduce on MR historyserver. As my understanding historyserver is used
> for recovery, so it should be OK without historyserver, right ?
>
>
> 2015-01-16 18:38:48,642 [main] INFO
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline
> service address: http://0.0.0.0:8188/ws/v1/timeline/
> 2015-01-16 18:38:48,643 [main] INFO org.apache.hadoop.yarn.client.RMProxy
> - Connecting to ResourceManager at /0.0.0.0:8032
> 2015-01-16 18:38:48,650 [main] INFO
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
> 2015-01-16 18:38:49,653 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 0
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2015-01-16 18:38:50,656 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 1
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2015-01-16 18:38:51,658 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 2
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
> 2015-01-16 18:38:52,661 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/0.0.0.0:10020. Already tried 3
> time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS)
>
>
>
>