You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Matt K <ma...@gmail.com> on 2015/01/23 19:41:59 UTC
yarn jobhistory server not displaying all jobs
Hello,
I am an issue with Yarn's JobHistory Server, which is making it painful to
debug jobs. The latest jobs (from the last 12 hours or so) are missing from
the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing
8 jobs only in the JobHistory, and 15 in Yarn UI.
Not much useful stuff in the logs. Every few hours, this exception pops up
in mapred-hadoop-historyserver.log, but I don't know if it's related.
2015-01-23 03:41:40,003 WARN
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
job files
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
User: hadoop is not allowed to impersonate hadoop
at org.apache.hadoop.ipc.Client.call(Client.java:1409)
at org.apache.hadoop.ipc.Client.call(Client.java:1362)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
at
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
at
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
at
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
at
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at
org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Has anyone ran into this before?
Thanks,
-Matt
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
I found the root cause. Sharing in case someone else runs into this issue.
I'm running Yarn, Hadoop 2.3.
The reason the jobs weren’t showing up in JobHistoryServer had to do with
how we submit jobs. If the same job is submitted via “hadoop jar …”
everything works fine. But if the job is submitted via “java –cp … “ which
is what we are doing, the job runs fine and all, but doesn’t make it to
JobHistoryServer.
The difference there is the classpath. When I added `hadoop classpath` to
our class path, the jobs started to show up.
There is definitely a bug in error handling, since there were no errors or
warnings in any of hadoop logs, but clearly there's a class required on the
client-side that was missing. I haven't tried tracking down the missing
jar/class.
-Matt
On Tue, Jan 27, 2015 at 9:39 AM, Matt K <ma...@gmail.com> wrote:
> Thanks Ravi! This helps.
>
> On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
>
>> Hi Matt!
>>
>> Take a look at the mapreduce.jobhistory.* configuration parameters here
>> for the delay in moving finished jobs to the HistoryServer:
>>
>> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
>> tried configuring hadoop proxy users
>>
>>
>> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
>> wrote:
>>
>>
>> Hello,
>>
>> I am an issue with Yarn's JobHistory Server, which is making it painful
>> to debug jobs. The latest jobs (from the last 12 hours or so) are missing
>> from the JobHistory Server, but present in ResourceManager Yarn UI. I am
>> seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
>>
>> Not much useful stuff in the logs. Every few hours, this exception pops
>> up in mapred-hadoop-historyserver.log, but I don't know if it's related.
>>
>> 2015-01-23 03:41:40,003 WARN
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
>> job files
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>> User: hadoop is not allowed to impersonate hadoop
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
>> at
>> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> Has anyone ran into this before?
>>
>> Thanks,
>> -Matt
>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
I found the root cause. Sharing in case someone else runs into this issue.
I'm running Yarn, Hadoop 2.3.
The reason the jobs weren’t showing up in JobHistoryServer had to do with
how we submit jobs. If the same job is submitted via “hadoop jar …”
everything works fine. But if the job is submitted via “java –cp … “ which
is what we are doing, the job runs fine and all, but doesn’t make it to
JobHistoryServer.
The difference there is the classpath. When I added `hadoop classpath` to
our class path, the jobs started to show up.
There is definitely a bug in error handling, since there were no errors or
warnings in any of hadoop logs, but clearly there's a class required on the
client-side that was missing. I haven't tried tracking down the missing
jar/class.
-Matt
On Tue, Jan 27, 2015 at 9:39 AM, Matt K <ma...@gmail.com> wrote:
> Thanks Ravi! This helps.
>
> On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
>
>> Hi Matt!
>>
>> Take a look at the mapreduce.jobhistory.* configuration parameters here
>> for the delay in moving finished jobs to the HistoryServer:
>>
>> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
>> tried configuring hadoop proxy users
>>
>>
>> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
>> wrote:
>>
>>
>> Hello,
>>
>> I am an issue with Yarn's JobHistory Server, which is making it painful
>> to debug jobs. The latest jobs (from the last 12 hours or so) are missing
>> from the JobHistory Server, but present in ResourceManager Yarn UI. I am
>> seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
>>
>> Not much useful stuff in the logs. Every few hours, this exception pops
>> up in mapred-hadoop-historyserver.log, but I don't know if it's related.
>>
>> 2015-01-23 03:41:40,003 WARN
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
>> job files
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>> User: hadoop is not allowed to impersonate hadoop
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
>> at
>> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> Has anyone ran into this before?
>>
>> Thanks,
>> -Matt
>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
I found the root cause. Sharing in case someone else runs into this issue.
I'm running Yarn, Hadoop 2.3.
The reason the jobs weren’t showing up in JobHistoryServer had to do with
how we submit jobs. If the same job is submitted via “hadoop jar …”
everything works fine. But if the job is submitted via “java –cp … “ which
is what we are doing, the job runs fine and all, but doesn’t make it to
JobHistoryServer.
The difference there is the classpath. When I added `hadoop classpath` to
our class path, the jobs started to show up.
There is definitely a bug in error handling, since there were no errors or
warnings in any of hadoop logs, but clearly there's a class required on the
client-side that was missing. I haven't tried tracking down the missing
jar/class.
-Matt
On Tue, Jan 27, 2015 at 9:39 AM, Matt K <ma...@gmail.com> wrote:
> Thanks Ravi! This helps.
>
> On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
>
>> Hi Matt!
>>
>> Take a look at the mapreduce.jobhistory.* configuration parameters here
>> for the delay in moving finished jobs to the HistoryServer:
>>
>> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
>> tried configuring hadoop proxy users
>>
>>
>> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
>> wrote:
>>
>>
>> Hello,
>>
>> I am an issue with Yarn's JobHistory Server, which is making it painful
>> to debug jobs. The latest jobs (from the last 12 hours or so) are missing
>> from the JobHistory Server, but present in ResourceManager Yarn UI. I am
>> seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
>>
>> Not much useful stuff in the logs. Every few hours, this exception pops
>> up in mapred-hadoop-historyserver.log, but I don't know if it's related.
>>
>> 2015-01-23 03:41:40,003 WARN
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
>> job files
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>> User: hadoop is not allowed to impersonate hadoop
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
>> at
>> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> Has anyone ran into this before?
>>
>> Thanks,
>> -Matt
>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
I found the root cause. Sharing in case someone else runs into this issue.
I'm running Yarn, Hadoop 2.3.
The reason the jobs weren’t showing up in JobHistoryServer had to do with
how we submit jobs. If the same job is submitted via “hadoop jar …”
everything works fine. But if the job is submitted via “java –cp … “ which
is what we are doing, the job runs fine and all, but doesn’t make it to
JobHistoryServer.
The difference there is the classpath. When I added `hadoop classpath` to
our class path, the jobs started to show up.
There is definitely a bug in error handling, since there were no errors or
warnings in any of hadoop logs, but clearly there's a class required on the
client-side that was missing. I haven't tried tracking down the missing
jar/class.
-Matt
On Tue, Jan 27, 2015 at 9:39 AM, Matt K <ma...@gmail.com> wrote:
> Thanks Ravi! This helps.
>
> On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
>
>> Hi Matt!
>>
>> Take a look at the mapreduce.jobhistory.* configuration parameters here
>> for the delay in moving finished jobs to the HistoryServer:
>>
>> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>>
>> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
>> tried configuring hadoop proxy users
>>
>>
>> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
>> wrote:
>>
>>
>> Hello,
>>
>> I am an issue with Yarn's JobHistory Server, which is making it painful
>> to debug jobs. The latest jobs (from the last 12 hours or so) are missing
>> from the JobHistory Server, but present in ResourceManager Yarn UI. I am
>> seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
>>
>> Not much useful stuff in the logs. Every few hours, this exception pops
>> up in mapred-hadoop-historyserver.log, but I don't know if it's related.
>>
>> 2015-01-23 03:41:40,003 WARN
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
>> job files
>> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
>> User: hadoop is not allowed to impersonate hadoop
>> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
>> at
>> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
>> at
>> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
>> at
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at
>> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>>
>> Has anyone ran into this before?
>>
>> Thanks,
>> -Matt
>>
>>
>>
>
>
> --
> www.calcmachine.com - easy online calculator.
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
Thanks Ravi! This helps.
On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
> Hi Matt!
>
> Take a look at the mapreduce.jobhistory.* configuration parameters here
> for the delay in moving finished jobs to the HistoryServer:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
> tried configuring hadoop proxy users
>
>
> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
> wrote:
>
>
> Hello,
>
> I am an issue with Yarn's JobHistory Server, which is making it painful to
> debug jobs. The latest jobs (from the last 12 hours or so) are missing from
> the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing
> 8 jobs only in the JobHistory, and 15 in Yarn UI.
>
> Not much useful stuff in the logs. Every few hours, this exception pops up
> in mapred-hadoop-historyserver.log, but I don't know if it's related.
>
> 2015-01-23 03:41:40,003 WARN
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
> job files
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: hadoop is not allowed to impersonate hadoop
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> Has anyone ran into this before?
>
> Thanks,
> -Matt
>
>
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
Thanks Ravi! This helps.
On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
> Hi Matt!
>
> Take a look at the mapreduce.jobhistory.* configuration parameters here
> for the delay in moving finished jobs to the HistoryServer:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
> tried configuring hadoop proxy users
>
>
> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
> wrote:
>
>
> Hello,
>
> I am an issue with Yarn's JobHistory Server, which is making it painful to
> debug jobs. The latest jobs (from the last 12 hours or so) are missing from
> the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing
> 8 jobs only in the JobHistory, and 15 in Yarn UI.
>
> Not much useful stuff in the logs. Every few hours, this exception pops up
> in mapred-hadoop-historyserver.log, but I don't know if it's related.
>
> 2015-01-23 03:41:40,003 WARN
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
> job files
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: hadoop is not allowed to impersonate hadoop
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> Has anyone ran into this before?
>
> Thanks,
> -Matt
>
>
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
Thanks Ravi! This helps.
On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
> Hi Matt!
>
> Take a look at the mapreduce.jobhistory.* configuration parameters here
> for the delay in moving finished jobs to the HistoryServer:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
> tried configuring hadoop proxy users
>
>
> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
> wrote:
>
>
> Hello,
>
> I am an issue with Yarn's JobHistory Server, which is making it painful to
> debug jobs. The latest jobs (from the last 12 hours or so) are missing from
> the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing
> 8 jobs only in the JobHistory, and 15 in Yarn UI.
>
> Not much useful stuff in the logs. Every few hours, this exception pops up
> in mapred-hadoop-historyserver.log, but I don't know if it's related.
>
> 2015-01-23 03:41:40,003 WARN
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
> job files
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: hadoop is not allowed to impersonate hadoop
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> Has anyone ran into this before?
>
> Thanks,
> -Matt
>
>
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Matt K <ma...@gmail.com>.
Thanks Ravi! This helps.
On Mon, Jan 26, 2015 at 2:22 PM, Ravi Prakash <ra...@ymail.com> wrote:
> Hi Matt!
>
> Take a look at the mapreduce.jobhistory.* configuration parameters here
> for the delay in moving finished jobs to the HistoryServer:
>
> https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
>
> I've seen this error "hadoop is not allowed to impersonate hadoop" when I
> tried configuring hadoop proxy users
>
>
> On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com>
> wrote:
>
>
> Hello,
>
> I am an issue with Yarn's JobHistory Server, which is making it painful to
> debug jobs. The latest jobs (from the last 12 hours or so) are missing from
> the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing
> 8 jobs only in the JobHistory, and 15 in Yarn UI.
>
> Not much useful stuff in the logs. Every few hours, this exception pops up
> in mapred-hadoop-historyserver.log, but I don't know if it's related.
>
> 2015-01-23 03:41:40,003 WARN
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process
> job files
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> User: hadoop is not allowed to impersonate hadoop
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at org.apache.hadoop.ipc.Client.call(Client.java:1362)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219)
> at
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127)
> at
> org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at
> org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
> Has anyone ran into this before?
>
> Thanks,
> -Matt
>
>
>
--
www.calcmachine.com - easy online calculator.
Re: yarn jobhistory server not displaying all jobs
Posted by Ravi Prakash <ra...@ymail.com>.
Hi Matt!
Take a look at the mapreduce.jobhistory.* configuration parameters here for the delay in moving finished jobs to the HistoryServer:https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
I've seen this error "hadoop is not allowed to impersonate hadoop" when I tried configuring hadoop proxy users
On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com> wrote:
Hello,
I am an issue with Yarn's JobHistory Server, which is making it painful to debug jobs. The latest jobs (from the last 12 hours or so) are missing from the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
Not much useful stuff in the logs. Every few hours, this exception pops up in mapred-hadoop-historyserver.log, but I don't know if it's related.
2015-01-23 03:41:40,003 WARN org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process job filesorg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Has anyone ran into this before?
Thanks,-Matt
Re: yarn jobhistory server not displaying all jobs
Posted by Ravi Prakash <ra...@ymail.com>.
Hi Matt!
Take a look at the mapreduce.jobhistory.* configuration parameters here for the delay in moving finished jobs to the HistoryServer:https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
I've seen this error "hadoop is not allowed to impersonate hadoop" when I tried configuring hadoop proxy users
On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com> wrote:
Hello,
I am an issue with Yarn's JobHistory Server, which is making it painful to debug jobs. The latest jobs (from the last 12 hours or so) are missing from the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
Not much useful stuff in the logs. Every few hours, this exception pops up in mapred-hadoop-historyserver.log, but I don't know if it's related.
2015-01-23 03:41:40,003 WARN org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process job filesorg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Has anyone ran into this before?
Thanks,-Matt
Re: yarn jobhistory server not displaying all jobs
Posted by Ravi Prakash <ra...@ymail.com>.
Hi Matt!
Take a look at the mapreduce.jobhistory.* configuration parameters here for the delay in moving finished jobs to the HistoryServer:https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
I've seen this error "hadoop is not allowed to impersonate hadoop" when I tried configuring hadoop proxy users
On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com> wrote:
Hello,
I am an issue with Yarn's JobHistory Server, which is making it painful to debug jobs. The latest jobs (from the last 12 hours or so) are missing from the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
Not much useful stuff in the logs. Every few hours, this exception pops up in mapred-hadoop-historyserver.log, but I don't know if it's related.
2015-01-23 03:41:40,003 WARN org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process job filesorg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Has anyone ran into this before?
Thanks,-Matt
Re: yarn jobhistory server not displaying all jobs
Posted by Ravi Prakash <ra...@ymail.com>.
Hi Matt!
Take a look at the mapreduce.jobhistory.* configuration parameters here for the delay in moving finished jobs to the HistoryServer:https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
I've seen this error "hadoop is not allowed to impersonate hadoop" when I tried configuring hadoop proxy users
On Friday, January 23, 2015 10:43 AM, Matt K <ma...@gmail.com> wrote:
Hello,
I am an issue with Yarn's JobHistory Server, which is making it painful to debug jobs. The latest jobs (from the last 12 hours or so) are missing from the JobHistory Server, but present in ResourceManager Yarn UI. I am seeing 8 jobs only in the JobHistory, and 15 in Yarn UI.
Not much useful stuff in the logs. Every few hours, this exception pops up in mapred-hadoop-historyserver.log, but I don't know if it's related.
2015-01-23 03:41:40,003 WARN org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService: Could not process job filesorg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: hadoop is not allowed to impersonate hadoop at org.apache.hadoop.ipc.Client.call(Client.java:1409) at org.apache.hadoop.ipc.Client.call(Client.java:1362) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:219) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1137) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1127) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1117) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:264) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:231) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:224) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1290) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:296) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.buildJobIndexInfo(KilledHistoryService.java:196) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$100(KilledHistoryService.java:85) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:128) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505)
Has anyone ran into this before?
Thanks,-Matt