You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Azuryy Yu <az...@gmail.com> on 2013/12/13 12:31:50 UTC
Container authenticate failed after YARN-961
Hi Dears,
After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job now.
I am using trunk in the test unsecure cluster, when submit Job, it
complained:
386931155525_0001_000002. Got exception: java.io.IOException: Failed on
local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]; Host Details : local host is: "
10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1410)
at org.apache.hadoop.ipc.Client.call(Client.java:1359)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Re: Container authenticate failed after YARN-961
Posted by Azuryy Yu <az...@gmail.com>.
Resolved. It's my fault, I changed some code from the trunk, which leads to
Token cannot generated.
On Sat, Dec 14, 2013 at 12:35 PM, Azuryy Yu <az...@gmail.com> wrote:
> HI,
>
>
> I looked at log carefully, then I found:
> 2013-12-14 12:29:19,184 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: KILL_CONTAINER sent to absent container
> container_1386995306465_0001_02_000001
> 2013-12-14 12:29:19,185 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1386995306465_0001
>
> It sounds like AMLauncher request container for AM, but node manager
> cannot get container.
>
> so how to solve it?
>
>
> On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> Maybe My yarn was configured incorrect.
>> in the NM log:
>> 2013-12-14 10:11:48,660 DEBUG
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
>> health-status : true,
>> 2013-12-14 10:11:48,660 DEBUG
>> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
>> test.slave152.com:8021 sending out status for 0 containers
>> 2013-12-14 10:11:48,687 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Event EventType: KILL_CONTAINER sent to absent container
>> container_1386987056408_0001_02_000001
>> 2013-12-14 10:11:48,688 WARN
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>> Event EventType: FINISH_APPLICATION sent to absent application
>> application_1386987056408_0001
>>
>> Does that mean there are 0 containers on this NM?
>>
>>
>>
>> On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> Hi Vinod,
>>> I enabled DEBUG log on both RM and NM.
>>> RM Log:
>>>
>>> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Sending sasl message state: NEGOTIATE
>>>
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Received SASL message state: NEGOTIATE
>>> auths {
>>> method: "TOKEN"
>>> mechanism: "DIGEST-MD5"
>>> protocol: ""
>>> serverId: "default"
>>> challenge:
>>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>>> }
>>>
>>> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
>>> Get token info proto:interface
>>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
>>> 2013-12-14 09:17:47,216 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
>>> cause:org.apache.hadoop.security.AccessControlException: Client cannot
>>> authenticate via:[TOKEN]
>>>
>>> NM Log:
>>> 2013-12-14 09:17:47,104 DEBUG
>>> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
>>> (auth:SIMPLE)
>>> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
>>> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
>>> Created SASL server with mechanism = DIGEST-MD5
>>> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
>>> message state: NEGOTIATE
>>> auths {
>>> method: "TOKEN"
>>> mechanism: "DIGEST-MD5"
>>> protocol: ""
>>> serverId: "default"
>>> challenge:
>>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>>> }
>>>
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>>> Wrote 166 bytes.
>>> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket
>>> Reader #1 for port 8021: disconnecting client 10.2.8.148:34175. Number
>>> of active connections: 0
>>>
>>> I am using HA based on trunk, there is in the log as:
>>> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
>>> /home/hadoop/dn_socket
>>> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token
>>> found for logical URI hdfs://test-cluster
>>>
>>> Please take a look, Thanks very much.
>>>
>>>
>>>
>>> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <az...@gmail.com> wrote:
>>>
>>>> Thanks Vinod,
>>>>
>>>> I am not chang MR AppMaster. I'll enable debug logging then paste
>>>> here.
>>>>
>>>>
>>>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>>>> vinodkv@hortonworks.com> wrote:
>>>>
>>>>> Is it stock MR or your modified MR AppMaster?
>>>>>
>>>>> Can you enable debug logging on NodeManager as well as the MR
>>>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>>>> based auth by default, so you should look if AM has the NMToken or not.
>>>>> Also check the NM logs.
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
>>>>>
>>>>> > Hi Dears,
>>>>> >
>>>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce
>>>>> Job now.
>>>>> >
>>>>> > I am using trunk in the test unsecure cluster, when submit Job, it
>>>>> > complained:
>>>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed
>>>>> on
>>>>> > local exception: java.io.IOException:
>>>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>>>> > at
>>>>> >
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>>>> > at
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> > at
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> > at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>>
>>>>> --
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to
>>>>> which it is addressed and may contain information that is confidential,
>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>> reader
>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>> and delete it from your system. Thank You.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Container authenticate failed after YARN-961
Posted by Azuryy Yu <az...@gmail.com>.
HI,
I looked at log carefully, then I found:
2013-12-14 12:29:19,184 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
container_1386995306465_0001_02_000001
2013-12-14 12:29:19,185 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: FINISH_APPLICATION sent to absent application
application_1386995306465_0001
It sounds like AMLauncher request container for AM, but node manager
cannot get container.
so how to solve it?
On Sat, Dec 14, 2013 at 10:20 AM, Azuryy Yu <az...@gmail.com> wrote:
> Maybe My yarn was configured incorrect.
> in the NM log:
> 2013-12-14 10:11:48,660 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
> health-status : true,
> 2013-12-14 10:11:48,660 DEBUG
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
> test.slave152.com:8021 sending out status for 0 containers
> 2013-12-14 10:11:48,687 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: KILL_CONTAINER sent to absent container
> container_1386987056408_0001_02_000001
> 2013-12-14 10:11:48,688 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
> Event EventType: FINISH_APPLICATION sent to absent application
> application_1386987056408_0001
>
> Does that mean there are 0 containers on this NM?
>
>
>
> On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> Hi Vinod,
>> I enabled DEBUG log on both RM and NM.
>> RM Log:
>>
>> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Sending sasl message state: NEGOTIATE
>>
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Received SASL message state: NEGOTIATE
>> auths {
>> method: "TOKEN"
>> mechanism: "DIGEST-MD5"
>> protocol: ""
>> serverId: "default"
>> challenge:
>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>> }
>>
>> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
>> Get token info proto:interface
>> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
>> 2013-12-14 09:17:47,216 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
>> cause:org.apache.hadoop.security.AccessControlException: Client cannot
>> authenticate via:[TOKEN]
>>
>> NM Log:
>> 2013-12-14 09:17:47,104 DEBUG
>> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
>> (auth:SIMPLE)
>> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
>> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
>> Created SASL server with mechanism = DIGEST-MD5
>> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
>> message state: NEGOTIATE
>> auths {
>> method: "TOKEN"
>> mechanism: "DIGEST-MD5"
>> protocol: ""
>> serverId: "default"
>> challenge:
>> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
>> }
>>
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
>> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
>> Wrote 166 bytes.
>> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader
>> #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of
>> active connections: 0
>>
>> I am using HA based on trunk, there is in the log as:
>> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
>> /home/hadoop/dn_socket
>> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token
>> found for logical URI hdfs://test-cluster
>>
>> Please take a look, Thanks very much.
>>
>>
>>
>> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>>> Thanks Vinod,
>>>
>>> I am not chang MR AppMaster. I'll enable debug logging then paste here.
>>>
>>>
>>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>> Is it stock MR or your modified MR AppMaster?
>>>>
>>>> Can you enable debug logging on NodeManager as well as the MR
>>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>>> based auth by default, so you should look if AM has the NMToken or not.
>>>> Also check the NM logs.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
>>>>
>>>> > Hi Dears,
>>>> >
>>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
>>>> now.
>>>> >
>>>> > I am using trunk in the test unsecure cluster, when submit Job, it
>>>> > complained:
>>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed
>>>> on
>>>> > local exception: java.io.IOException:
>>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>>> > at
>>>> >
>>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>>> > at
>>>> >
>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>>> > at
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> > at
>>>> >
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> > at java.lang.Thread.run(Thread.java:744)
>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>>
>>
>
Re: Container authenticate failed after YARN-961
Posted by Azuryy Yu <az...@gmail.com>.
Maybe My yarn was configured incorrect.
in the NM log:
2013-12-14 10:11:48,660 DEBUG
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node's
health-status : true,
2013-12-14 10:11:48,660 DEBUG
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl:
test.slave152.com:8021 sending out status for 0 containers
2013-12-14 10:11:48,687 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
container_1386987056408_0001_02_000001
2013-12-14 10:11:48,688 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: FINISH_APPLICATION sent to absent application
application_1386987056408_0001
Does that mean there are 0 containers on this NM?
On Sat, Dec 14, 2013 at 9:38 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Vinod,
> I enabled DEBUG log on both RM and NM.
> RM Log:
>
> 2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Sending sasl message state: NEGOTIATE
>
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Received SASL message state: NEGOTIATE
> auths {
> method: "TOKEN"
> mechanism: "DIGEST-MD5"
> protocol: ""
> serverId: "default"
> challenge:
> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
> }
>
> 2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient:
> Get token info proto:interface
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
> 2013-12-14 09:17:47,216 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN]
>
> NM Log:
> 2013-12-14 09:17:47,104 DEBUG
> org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
> (auth:SIMPLE)
> from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
> 2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
> Created SASL server with mechanism = DIGEST-MD5
> 2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
> message state: NEGOTIATE
> auths {
> method: "TOKEN"
> mechanism: "DIGEST-MD5"
> protocol: ""
> serverId: "default"
> challenge:
> "realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
> }
>
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
> 2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
> Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
> Wrote 166 bytes.
> 2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader
> #1 for port 8021: disconnecting client 10.2.8.148:34175. Number of active
> connections: 0
>
> I am using HA based on trunk, there is in the log as:
> 13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
> /home/hadoop/dn_socket
> 13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token found
> for logical URI hdfs://test-cluster
>
> Please take a look, Thanks very much.
>
>
>
> On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <az...@gmail.com> wrote:
>
>> Thanks Vinod,
>>
>> I am not chang MR AppMaster. I'll enable debug logging then paste here.
>>
>>
>> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>> Is it stock MR or your modified MR AppMaster?
>>>
>>> Can you enable debug logging on NodeManager as well as the MR
>>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>>> based auth by default, so you should look if AM has the NMToken or not.
>>> Also check the NM logs.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
>>>
>>> > Hi Dears,
>>> >
>>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
>>> now.
>>> >
>>> > I am using trunk in the test unsecure cluster, when submit Job, it
>>> > complained:
>>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed on
>>> > local exception: java.io.IOException:
>>> > org.apache.hadoop.security.AccessControlException: Client cannot
>>> > authenticate via:[TOKEN]; Host Details : local host is: "
>>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>>> > at
>>> >
>>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>>> > at
>>> >
>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> > at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> > at java.lang.Thread.run(Thread.java:744)
>>>
>>>
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>>> immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>
Re: Container authenticate failed after YARN-961
Posted by Azuryy Yu <az...@gmail.com>.
Hi Vinod,
I enabled DEBUG log on both RM and NM.
RM Log:
2013-12-14 09:17:47,093 DEBUG org.apache.hadoop.security.SaslRpcClient:
Sending sasl message state: NEGOTIATE
2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.security.SaslRpcClient:
Received SASL message state: NEGOTIATE
auths {
method: "TOKEN"
mechanism: "DIGEST-MD5"
protocol: ""
serverId: "default"
challenge:
"realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
}
2013-12-14 09:17:47,216 DEBUG org.apache.hadoop.security.SaslRpcClient: Get
token info proto:interface
org.apache.hadoop.yarn.api.ContainerManagementProtocolPB info:null
2013-12-14 09:17:47,216 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:appattempt_1386983771970_0001_000001 (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]
NM Log:
2013-12-14 09:17:47,104 DEBUG
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hadoop
(auth:SIMPLE)
from:org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:150)
2013-12-14 09:17:47,111 DEBUG org.apache.hadoop.security.SaslRpcServer:
Created SASL server with mechanism = DIGEST-MD5
2013-12-14 09:17:47,205 DEBUG org.apache.hadoop.ipc.Server: Sending sasl
message state: NEGOTIATE
auths {
method: "TOKEN"
mechanism: "DIGEST-MD5"
protocol: ""
serverId: "default"
challenge:
"realm=\"default\",nonce=\"7+Tb5yJJQV0Mpd6r2OwezSWoeN07h7E0DelSSJJi\",qop=\"auth\",charset=utf-8,algorithm=md5-sess"
}
2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1
2013-12-14 09:17:47,208 DEBUG org.apache.hadoop.ipc.Server: IPC Server
Responder: responding to null from 10.2.8.148:34175 Call#-33 Retry#-1 Wrote
166 bytes.
2013-12-14 09:17:47,223 DEBUG org.apache.hadoop.ipc.Server: Socket Reader
#1 for port 8021: disconnecting client 10.2.8.148:34175. Number of active
connections: 0
I am using HA based on trunk, there is in the log as:
13/12/14 09:17:42 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path =
/home/hadoop/dn_socket
13/12/14 09:17:42 DEBUG hdfs.HAUtil: No HA service delegation token found
for logical URI hdfs://test-cluster
Please take a look, Thanks very much.
On Sat, Dec 14, 2013 at 8:56 AM, Azuryy Yu <az...@gmail.com> wrote:
> Thanks Vinod,
>
> I am not chang MR AppMaster. I'll enable debug logging then paste here.
>
>
> On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>> Is it stock MR or your modified MR AppMaster?
>>
>> Can you enable debug logging on NodeManager as well as the MR
>> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
>> based auth by default, so you should look if AM has the NMToken or not.
>> Also check the NM logs.
>>
>> Thanks,
>> +Vinod
>>
>> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
>>
>> > Hi Dears,
>> >
>> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
>> now.
>> >
>> > I am using trunk in the test unsecure cluster, when submit Job, it
>> > complained:
>> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed on
>> > local exception: java.io.IOException:
>> > org.apache.hadoop.security.AccessControlException: Client cannot
>> > authenticate via:[TOKEN]; Host Details : local host is: "
>> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
>> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
>> > at
>> >
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>> > at
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
>> > at
>> >
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> > at java.lang.Thread.run(Thread.java:744)
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>
Re: Container authenticate failed after YARN-961
Posted by Azuryy Yu <az...@gmail.com>.
Thanks Vinod,
I am not chang MR AppMaster. I'll enable debug logging then paste here.
On Sat, Dec 14, 2013 at 2:36 AM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:
> Is it stock MR or your modified MR AppMaster?
>
> Can you enable debug logging on NodeManager as well as the MR
> ApplicationMaster and trace what exactly is happening? We enabled TOKEN
> based auth by default, so you should look if AM has the NMToken or not.
> Also check the NM logs.
>
> Thanks,
> +Vinod
>
> On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
>
> > Hi Dears,
> >
> > After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job
> now.
> >
> > I am using trunk in the test unsecure cluster, when submit Job, it
> > complained:
> > 386931155525_0001_000002. Got exception: java.io.IOException: Failed on
> > local exception: java.io.IOException:
> > org.apache.hadoop.security.AccessControlException: Client cannot
> > authenticate via:[TOKEN]; Host Details : local host is: "
> > 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
> > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> > at org.apache.hadoop.ipc.Client.call(Client.java:1359)
> > at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> > at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
> > at
> >
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
> > at
> >
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
> > at
> >
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:744)
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Container authenticate failed after YARN-961
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Is it stock MR or your modified MR AppMaster?
Can you enable debug logging on NodeManager as well as the MR ApplicationMaster and trace what exactly is happening? We enabled TOKEN based auth by default, so you should look if AM has the NMToken or not. Also check the NM logs.
Thanks,
+Vinod
On Dec 13, 2013, at 3:31 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Dears,
>
> After YARN-961 committed to the trunk, we cannot sumbit MapReduce Job now.
>
> I am using trunk in the test unsecure cluster, when submit Job, it
> complained:
> 386931155525_0001_000002. Got exception: java.io.IOException: Failed on
> local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN]; Host Details : local host is: "
> 10.2.8.148/10.2.8.148"; destination host is: "test.152.com":8021;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
> at org.apache.hadoop.ipc.Client.call(Client.java:1410)
> at org.apache.hadoop.ipc.Client.call(Client.java:1359)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy23.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
> at
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:249)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.