You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by rab ra <ra...@gmail.com> on 2013/11/13 13:05:00 UTC
Problem with RPC encryption over wire
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature
in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 & 3)
NodeManager
I am trying to make data transfer between master and slave secure. For
that, I wanted to use data encryption over wire (RPC encryption) feature of
hadoop 2.2.0
When I ran the code, I get the below exception
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read.
In another run, I saw in log the following error
No common protection layer between server and client
Not sure whether my configuration is inline with what I want to achieve.
Can someone give me some hint on where I am going wrong?
By the way, I have the below configuration setting in all of these nodes
Core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp</value>
</property>
<!--
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
-->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/dfs-2.2.0/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/dfs-2.2.0/data</value>
</property>
<property>
<name>dfs.encrypt.data.transfer</name>
<value>true</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>rc4</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<!--
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8032</value>
</property>
-->
<property>
<name>mapreduce.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024m</value>
</property>
</configuration>
Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
With thanks and regards
Rab
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Any hint ?
On 14 Nov 2013 11:45, "rab ra" <ra...@gmail.com> wrote:
> Thank for the response.
>
> I have removed rpc.protection parameter in all of my configuration and now
> I am getting an error as below:-
>
>
>
> Any Hint on whats going on here
>
>
>
> 13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
> attempt_1384339616944_0002_m_000026_0, Status : FAILED
>
> Container launch failed for container_1384339616944_0002_01_000029 :
> java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
> U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]; For more details see:
> http://wiki.apache.org/hadoop/SocketTimeout
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>
> at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]
>
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
>
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
>
>
>
> 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
>
> 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
>
> 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
>
> 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
>
> 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
>
> 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
>
> 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
>
> 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
>
> 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
>
> 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
>
> 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
>
> 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
>
> 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
>
> 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
>
> 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
>
> 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
>
> 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
>
> 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
>
> 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
>
> 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
>
> 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
>
> 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
> successfully
>
> 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 ERROR security.UserGroupInformation:
> PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
> cause:java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> [Error] Exception:
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
>
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
>
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
>
> at
> org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
>
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
>
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
>
> at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
>
>> "No common protection layer between server and client " likely means the
>> host for job submission does not have hadoop.rpc.protection=privacy. In
>> order for QOP to work, all client hosts (DN & others used to access the
>> cluster) must have an identical setting.
>>
>> A few quick questions: I'm assuming you mis-posted your configs and the
>> protection setting isn't really commented out? Your configs don't show
>> security being enabled, but you do have it enabled, correct? Otherwise QOP
>> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
>> changing the QOP? Since your defaultFS is just "master", are you using HA?
>>
>> It's a bit concerning that you aren't consistently receiving the
>> mismatch error. Is the client looping on retries and then you get timeouts
>> after 5 attempts? If yes, we've got a major bug. 5 is the default number
>> of RPC readers which handle SASL auth which means the protection mismatch
>> is killing off the reader threads and rendering the NN unusable. This
>> shouldn't be possible, but what does your NN log show?
>>
>> Daryn
>>
>> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>>
>>
>>
>> Hello,
>>
>> I am facing a problem in using Hadoop RPC encryption while transfer
>> feature in hadoop 2.2.0. I have 3 node cluster
>>
>>
>> Service running in node 1 (master)
>> Resource manager
>> Namenode
>> DataNode
>> SecondaryNamenode
>>
>> Service running in slaves ( node 2 & 3)
>> NodeManager
>>
>>
>>
>> I am trying to make data transfer between master and slave secure. For
>> that, I wanted to use data encryption over wire (RPC encryption) feature of
>> hadoop 2.2.0
>>
>> When I ran the code, I get the below exception
>>
>> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
>> waiting for channel to be ready for read.
>>
>>
>> In another run, I saw in log the following error
>>
>> No common protection layer between server and client
>>
>> Not sure whether my configuration is inline with what I want to achieve.
>>
>> Can someone give me some hint on where I am going wrong?
>>
>> By the way, I have the below configuration setting in all of these nodes
>>
>> Core-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>fs.defaultFS</name>
>> <value>hdfs://master:8020</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/tmp</value>
>> </property>
>> <!--
>> <property>
>> <name>hadoop.rpc.protection</name>
>> <value>privacy</value>
>> </property>
>> -->
>> <property>
>> <name>io.file.buffer.size</name>
>> <value>131072</value>
>> </property>
>>
>> </configuration>
>>
>> Hdfs-site.xml
>> <configuration>
>>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/name</value>
>> </property>
>>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/data</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer.algorithm</name>
>> <value>rc4</value>
>> </property>
>>
>> <property>
>> <name>dfs.block.access.token.enable</name>
>> <value>true</value>
>> </property>
>>
>> </configuration>
>>
>> Mapred-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <!--
>> <property>
>> <name>mapreduce.jobtracker.address</name>
>> <value>master:8032</value>
>> </property>
>> -->
>> <property>
>> <name>mapreduce.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.reduce.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.java.opts</name>
>> <value>-Xmx1024m</value>
>> </property>
>>
>> </configuration>
>>
>>
>> Yarn-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>yarn.resourcemanager.hostname</name>
>> <value>master</value>
>> </property>
>>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>> </configuration>
>>
>>
>>
>> With thanks and regards
>> Rab
>>
>>
>>
>>
>>
>>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Any hint ?
On 14 Nov 2013 11:45, "rab ra" <ra...@gmail.com> wrote:
> Thank for the response.
>
> I have removed rpc.protection parameter in all of my configuration and now
> I am getting an error as below:-
>
>
>
> Any Hint on whats going on here
>
>
>
> 13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
> attempt_1384339616944_0002_m_000026_0, Status : FAILED
>
> Container launch failed for container_1384339616944_0002_01_000029 :
> java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
> U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]; For more details see:
> http://wiki.apache.org/hadoop/SocketTimeout
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>
> at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]
>
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
>
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
>
>
>
> 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
>
> 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
>
> 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
>
> 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
>
> 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
>
> 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
>
> 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
>
> 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
>
> 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
>
> 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
>
> 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
>
> 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
>
> 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
>
> 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
>
> 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
>
> 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
>
> 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
>
> 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
>
> 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
>
> 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
>
> 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
>
> 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
> successfully
>
> 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 ERROR security.UserGroupInformation:
> PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
> cause:java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> [Error] Exception:
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
>
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
>
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
>
> at
> org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
>
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
>
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
>
> at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
>
>> "No common protection layer between server and client " likely means the
>> host for job submission does not have hadoop.rpc.protection=privacy. In
>> order for QOP to work, all client hosts (DN & others used to access the
>> cluster) must have an identical setting.
>>
>> A few quick questions: I'm assuming you mis-posted your configs and the
>> protection setting isn't really commented out? Your configs don't show
>> security being enabled, but you do have it enabled, correct? Otherwise QOP
>> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
>> changing the QOP? Since your defaultFS is just "master", are you using HA?
>>
>> It's a bit concerning that you aren't consistently receiving the
>> mismatch error. Is the client looping on retries and then you get timeouts
>> after 5 attempts? If yes, we've got a major bug. 5 is the default number
>> of RPC readers which handle SASL auth which means the protection mismatch
>> is killing off the reader threads and rendering the NN unusable. This
>> shouldn't be possible, but what does your NN log show?
>>
>> Daryn
>>
>> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>>
>>
>>
>> Hello,
>>
>> I am facing a problem in using Hadoop RPC encryption while transfer
>> feature in hadoop 2.2.0. I have 3 node cluster
>>
>>
>> Service running in node 1 (master)
>> Resource manager
>> Namenode
>> DataNode
>> SecondaryNamenode
>>
>> Service running in slaves ( node 2 & 3)
>> NodeManager
>>
>>
>>
>> I am trying to make data transfer between master and slave secure. For
>> that, I wanted to use data encryption over wire (RPC encryption) feature of
>> hadoop 2.2.0
>>
>> When I ran the code, I get the below exception
>>
>> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
>> waiting for channel to be ready for read.
>>
>>
>> In another run, I saw in log the following error
>>
>> No common protection layer between server and client
>>
>> Not sure whether my configuration is inline with what I want to achieve.
>>
>> Can someone give me some hint on where I am going wrong?
>>
>> By the way, I have the below configuration setting in all of these nodes
>>
>> Core-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>fs.defaultFS</name>
>> <value>hdfs://master:8020</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/tmp</value>
>> </property>
>> <!--
>> <property>
>> <name>hadoop.rpc.protection</name>
>> <value>privacy</value>
>> </property>
>> -->
>> <property>
>> <name>io.file.buffer.size</name>
>> <value>131072</value>
>> </property>
>>
>> </configuration>
>>
>> Hdfs-site.xml
>> <configuration>
>>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/name</value>
>> </property>
>>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/data</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer.algorithm</name>
>> <value>rc4</value>
>> </property>
>>
>> <property>
>> <name>dfs.block.access.token.enable</name>
>> <value>true</value>
>> </property>
>>
>> </configuration>
>>
>> Mapred-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <!--
>> <property>
>> <name>mapreduce.jobtracker.address</name>
>> <value>master:8032</value>
>> </property>
>> -->
>> <property>
>> <name>mapreduce.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.reduce.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.java.opts</name>
>> <value>-Xmx1024m</value>
>> </property>
>>
>> </configuration>
>>
>>
>> Yarn-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>yarn.resourcemanager.hostname</name>
>> <value>master</value>
>> </property>
>>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>> </configuration>
>>
>>
>>
>> With thanks and regards
>> Rab
>>
>>
>>
>>
>>
>>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Any hint ?
On 14 Nov 2013 11:45, "rab ra" <ra...@gmail.com> wrote:
> Thank for the response.
>
> I have removed rpc.protection parameter in all of my configuration and now
> I am getting an error as below:-
>
>
>
> Any Hint on whats going on here
>
>
>
> 13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
> attempt_1384339616944_0002_m_000026_0, Status : FAILED
>
> Container launch failed for container_1384339616944_0002_01_000029 :
> java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
> U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]; For more details see:
> http://wiki.apache.org/hadoop/SocketTimeout
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>
> at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]
>
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
>
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
>
>
>
> 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
>
> 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
>
> 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
>
> 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
>
> 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
>
> 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
>
> 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
>
> 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
>
> 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
>
> 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
>
> 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
>
> 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
>
> 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
>
> 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
>
> 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
>
> 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
>
> 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
>
> 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
>
> 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
>
> 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
>
> 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
>
> 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
> successfully
>
> 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 ERROR security.UserGroupInformation:
> PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
> cause:java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> [Error] Exception:
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
>
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
>
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
>
> at
> org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
>
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
>
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
>
> at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
>
>> "No common protection layer between server and client " likely means the
>> host for job submission does not have hadoop.rpc.protection=privacy. In
>> order for QOP to work, all client hosts (DN & others used to access the
>> cluster) must have an identical setting.
>>
>> A few quick questions: I'm assuming you mis-posted your configs and the
>> protection setting isn't really commented out? Your configs don't show
>> security being enabled, but you do have it enabled, correct? Otherwise QOP
>> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
>> changing the QOP? Since your defaultFS is just "master", are you using HA?
>>
>> It's a bit concerning that you aren't consistently receiving the
>> mismatch error. Is the client looping on retries and then you get timeouts
>> after 5 attempts? If yes, we've got a major bug. 5 is the default number
>> of RPC readers which handle SASL auth which means the protection mismatch
>> is killing off the reader threads and rendering the NN unusable. This
>> shouldn't be possible, but what does your NN log show?
>>
>> Daryn
>>
>> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>>
>>
>>
>> Hello,
>>
>> I am facing a problem in using Hadoop RPC encryption while transfer
>> feature in hadoop 2.2.0. I have 3 node cluster
>>
>>
>> Service running in node 1 (master)
>> Resource manager
>> Namenode
>> DataNode
>> SecondaryNamenode
>>
>> Service running in slaves ( node 2 & 3)
>> NodeManager
>>
>>
>>
>> I am trying to make data transfer between master and slave secure. For
>> that, I wanted to use data encryption over wire (RPC encryption) feature of
>> hadoop 2.2.0
>>
>> When I ran the code, I get the below exception
>>
>> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
>> waiting for channel to be ready for read.
>>
>>
>> In another run, I saw in log the following error
>>
>> No common protection layer between server and client
>>
>> Not sure whether my configuration is inline with what I want to achieve.
>>
>> Can someone give me some hint on where I am going wrong?
>>
>> By the way, I have the below configuration setting in all of these nodes
>>
>> Core-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>fs.defaultFS</name>
>> <value>hdfs://master:8020</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/tmp</value>
>> </property>
>> <!--
>> <property>
>> <name>hadoop.rpc.protection</name>
>> <value>privacy</value>
>> </property>
>> -->
>> <property>
>> <name>io.file.buffer.size</name>
>> <value>131072</value>
>> </property>
>>
>> </configuration>
>>
>> Hdfs-site.xml
>> <configuration>
>>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/name</value>
>> </property>
>>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/data</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer.algorithm</name>
>> <value>rc4</value>
>> </property>
>>
>> <property>
>> <name>dfs.block.access.token.enable</name>
>> <value>true</value>
>> </property>
>>
>> </configuration>
>>
>> Mapred-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <!--
>> <property>
>> <name>mapreduce.jobtracker.address</name>
>> <value>master:8032</value>
>> </property>
>> -->
>> <property>
>> <name>mapreduce.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.reduce.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.java.opts</name>
>> <value>-Xmx1024m</value>
>> </property>
>>
>> </configuration>
>>
>>
>> Yarn-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>yarn.resourcemanager.hostname</name>
>> <value>master</value>
>> </property>
>>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>> </configuration>
>>
>>
>>
>> With thanks and regards
>> Rab
>>
>>
>>
>>
>>
>>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Any hint ?
On 14 Nov 2013 11:45, "rab ra" <ra...@gmail.com> wrote:
> Thank for the response.
>
> I have removed rpc.protection parameter in all of my configuration and now
> I am getting an error as below:-
>
>
>
> Any Hint on whats going on here
>
>
>
> 13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
> attempt_1384339616944_0002_m_000026_0, Status : FAILED
>
> Container launch failed for container_1384339616944_0002_01_000029 :
> java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
> U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]; For more details see:
> http://wiki.apache.org/hadoop/SocketTimeout
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
>
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1351)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
>
> at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
>
> at
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read. ch :
> java.nio.channels.SocketChannel[connected local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
> 40.221.95.97:45003]
>
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
>
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> at
> org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
>
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
>
>
>
> 13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
>
> 13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
>
> 13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
>
> 13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
>
> 13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
>
> 13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
>
> 13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
>
> 13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
>
> 13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
>
> 13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
>
> 13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
>
> 13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
>
> 13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
>
> 13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
>
> 13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
>
> 13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
>
> 13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
>
> 13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
>
> 13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
>
> 13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
>
> 13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
>
> 13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
> successfully
>
> 13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
> server
>
> 13/11/14 10:29:56 ERROR security.UserGroupInformation:
> PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
> cause:java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> [Error] Exception:
>
> java.io.IOException:
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
> java.lang.NullPointerException
>
> at
> org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
>
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
>
> at
> org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
>
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
>
> at
> org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
>
> at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
> at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
>
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
>
> at
> org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
>
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
>
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
>
> at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
>
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>
> at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
>
>> "No common protection layer between server and client " likely means the
>> host for job submission does not have hadoop.rpc.protection=privacy. In
>> order for QOP to work, all client hosts (DN & others used to access the
>> cluster) must have an identical setting.
>>
>> A few quick questions: I'm assuming you mis-posted your configs and the
>> protection setting isn't really commented out? Your configs don't show
>> security being enabled, but you do have it enabled, correct? Otherwise QOP
>> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
>> changing the QOP? Since your defaultFS is just "master", are you using HA?
>>
>> It's a bit concerning that you aren't consistently receiving the
>> mismatch error. Is the client looping on retries and then you get timeouts
>> after 5 attempts? If yes, we've got a major bug. 5 is the default number
>> of RPC readers which handle SASL auth which means the protection mismatch
>> is killing off the reader threads and rendering the NN unusable. This
>> shouldn't be possible, but what does your NN log show?
>>
>> Daryn
>>
>> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>>
>>
>>
>> Hello,
>>
>> I am facing a problem in using Hadoop RPC encryption while transfer
>> feature in hadoop 2.2.0. I have 3 node cluster
>>
>>
>> Service running in node 1 (master)
>> Resource manager
>> Namenode
>> DataNode
>> SecondaryNamenode
>>
>> Service running in slaves ( node 2 & 3)
>> NodeManager
>>
>>
>>
>> I am trying to make data transfer between master and slave secure. For
>> that, I wanted to use data encryption over wire (RPC encryption) feature of
>> hadoop 2.2.0
>>
>> When I ran the code, I get the below exception
>>
>> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
>> waiting for channel to be ready for read.
>>
>>
>> In another run, I saw in log the following error
>>
>> No common protection layer between server and client
>>
>> Not sure whether my configuration is inline with what I want to achieve.
>>
>> Can someone give me some hint on where I am going wrong?
>>
>> By the way, I have the below configuration setting in all of these nodes
>>
>> Core-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>fs.defaultFS</name>
>> <value>hdfs://master:8020</value>
>> </property>
>>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/tmp</value>
>> </property>
>> <!--
>> <property>
>> <name>hadoop.rpc.protection</name>
>> <value>privacy</value>
>> </property>
>> -->
>> <property>
>> <name>io.file.buffer.size</name>
>> <value>131072</value>
>> </property>
>>
>> </configuration>
>>
>> Hdfs-site.xml
>> <configuration>
>>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>dfs.name.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/name</value>
>> </property>
>>
>> <property>
>> <name>dfs.data.dir</name>
>> <value>/app/hadoop/dfs-2.2.0/data</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>dfs.encrypt.data.transfer.algorithm</name>
>> <value>rc4</value>
>> </property>
>>
>> <property>
>> <name>dfs.block.access.token.enable</name>
>> <value>true</value>
>> </property>
>>
>> </configuration>
>>
>> Mapred-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>mapreduce.framework.name</name>
>> <value>yarn</value>
>> </property>
>> <!--
>> <property>
>> <name>mapreduce.jobtracker.address</name>
>> <value>master:8032</value>
>> </property>
>> -->
>> <property>
>> <name>mapreduce.tasktracker.map.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>> <value>1</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.reduce.speculative</name>
>> <value>false</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.java.opts</name>
>> <value>-Xmx1024m</value>
>> </property>
>>
>> </configuration>
>>
>>
>> Yarn-site.xml
>>
>> <configuration>
>>
>> <property>
>> <name>yarn.resourcemanager.hostname</name>
>> <value>master</value>
>> </property>
>>
>> <property>
>> <name>yarn.log-aggregation-enable</name>
>> <value>true</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>> </configuration>
>>
>>
>>
>> With thanks and regards
>> Rab
>>
>>
>>
>>
>>
>>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Thank for the response.
I have removed rpc.protection parameter in all of my configuration and now
I am getting an error as below:-
Any Hint on whats going on here
13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
attempt_1384339616944_0002_m_000026_0, Status : FAILED
Container launch failed for container_1384339616944_0002_01_000029 :
java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
successfully
13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
[Error] Exception:
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
> "No common protection layer between server and client " likely means the
> host for job submission does not have hadoop.rpc.protection=privacy. In
> order for QOP to work, all client hosts (DN & others used to access the
> cluster) must have an identical setting.
>
> A few quick questions: I'm assuming you mis-posted your configs and the
> protection setting isn't really commented out? Your configs don't show
> security being enabled, but you do have it enabled, correct? Otherwise QOP
> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
> changing the QOP? Since your defaultFS is just "master", are you using HA?
>
> It's a bit concerning that you aren't consistently receiving the
> mismatch error. Is the client looping on retries and then you get timeouts
> after 5 attempts? If yes, we've got a major bug. 5 is the default number
> of RPC readers which handle SASL auth which means the protection mismatch
> is killing off the reader threads and rendering the NN unusable. This
> shouldn't be possible, but what does your NN log show?
>
> Daryn
>
> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>
>
>
> Hello,
>
> I am facing a problem in using Hadoop RPC encryption while transfer
> feature in hadoop 2.2.0. I have 3 node cluster
>
>
> Service running in node 1 (master)
> Resource manager
> Namenode
> DataNode
> SecondaryNamenode
>
> Service running in slaves ( node 2 & 3)
> NodeManager
>
>
>
> I am trying to make data transfer between master and slave secure. For
> that, I wanted to use data encryption over wire (RPC encryption) feature of
> hadoop 2.2.0
>
> When I ran the code, I get the below exception
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read.
>
>
> In another run, I saw in log the following error
>
> No common protection layer between server and client
>
> Not sure whether my configuration is inline with what I want to achieve.
>
> Can someone give me some hint on where I am going wrong?
>
> By the way, I have the below configuration setting in all of these nodes
>
> Core-site.xml
>
> <configuration>
>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://master:8020</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp</value>
> </property>
> <!--
> <property>
> <name>hadoop.rpc.protection</name>
> <value>privacy</value>
> </property>
> -->
> <property>
> <name>io.file.buffer.size</name>
> <value>131072</value>
> </property>
>
> </configuration>
>
> Hdfs-site.xml
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/app/hadoop/dfs-2.2.0/name</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/app/hadoop/dfs-2.2.0/data</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer.algorithm</name>
> <value>rc4</value>
> </property>
>
> <property>
> <name>dfs.block.access.token.enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Mapred-site.xml
>
> <configuration>
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <!--
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>master:8032</value>
> </property>
> -->
> <property>
> <name>mapreduce.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.map.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.reduce.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.map.java.opts</name>
> <value>-Xmx1024m</value>
> </property>
>
> </configuration>
>
>
> Yarn-site.xml
>
> <configuration>
>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>master</value>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
> </configuration>
>
>
>
> With thanks and regards
> Rab
>
>
>
>
>
>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Thank for the response.
I have removed rpc.protection parameter in all of my configuration and now
I am getting an error as below:-
Any Hint on whats going on here
13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
attempt_1384339616944_0002_m_000026_0, Status : FAILED
Container launch failed for container_1384339616944_0002_01_000029 :
java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
successfully
13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
[Error] Exception:
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
> "No common protection layer between server and client " likely means the
> host for job submission does not have hadoop.rpc.protection=privacy. In
> order for QOP to work, all client hosts (DN & others used to access the
> cluster) must have an identical setting.
>
> A few quick questions: I'm assuming you mis-posted your configs and the
> protection setting isn't really commented out? Your configs don't show
> security being enabled, but you do have it enabled, correct? Otherwise QOP
> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
> changing the QOP? Since your defaultFS is just "master", are you using HA?
>
> It's a bit concerning that you aren't consistently receiving the
> mismatch error. Is the client looping on retries and then you get timeouts
> after 5 attempts? If yes, we've got a major bug. 5 is the default number
> of RPC readers which handle SASL auth which means the protection mismatch
> is killing off the reader threads and rendering the NN unusable. This
> shouldn't be possible, but what does your NN log show?
>
> Daryn
>
> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>
>
>
> Hello,
>
> I am facing a problem in using Hadoop RPC encryption while transfer
> feature in hadoop 2.2.0. I have 3 node cluster
>
>
> Service running in node 1 (master)
> Resource manager
> Namenode
> DataNode
> SecondaryNamenode
>
> Service running in slaves ( node 2 & 3)
> NodeManager
>
>
>
> I am trying to make data transfer between master and slave secure. For
> that, I wanted to use data encryption over wire (RPC encryption) feature of
> hadoop 2.2.0
>
> When I ran the code, I get the below exception
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read.
>
>
> In another run, I saw in log the following error
>
> No common protection layer between server and client
>
> Not sure whether my configuration is inline with what I want to achieve.
>
> Can someone give me some hint on where I am going wrong?
>
> By the way, I have the below configuration setting in all of these nodes
>
> Core-site.xml
>
> <configuration>
>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://master:8020</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp</value>
> </property>
> <!--
> <property>
> <name>hadoop.rpc.protection</name>
> <value>privacy</value>
> </property>
> -->
> <property>
> <name>io.file.buffer.size</name>
> <value>131072</value>
> </property>
>
> </configuration>
>
> Hdfs-site.xml
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/app/hadoop/dfs-2.2.0/name</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/app/hadoop/dfs-2.2.0/data</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer.algorithm</name>
> <value>rc4</value>
> </property>
>
> <property>
> <name>dfs.block.access.token.enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Mapred-site.xml
>
> <configuration>
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <!--
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>master:8032</value>
> </property>
> -->
> <property>
> <name>mapreduce.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.map.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.reduce.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.map.java.opts</name>
> <value>-Xmx1024m</value>
> </property>
>
> </configuration>
>
>
> Yarn-site.xml
>
> <configuration>
>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>master</value>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
> </configuration>
>
>
>
> With thanks and regards
> Rab
>
>
>
>
>
>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Thank for the response.
I have removed rpc.protection parameter in all of my configuration and now
I am getting an error as below:-
Any Hint on whats going on here
13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
attempt_1384339616944_0002_m_000026_0, Status : FAILED
Container launch failed for container_1384339616944_0002_01_000029 :
java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
successfully
13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
[Error] Exception:
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
> "No common protection layer between server and client " likely means the
> host for job submission does not have hadoop.rpc.protection=privacy. In
> order for QOP to work, all client hosts (DN & others used to access the
> cluster) must have an identical setting.
>
> A few quick questions: I'm assuming you mis-posted your configs and the
> protection setting isn't really commented out? Your configs don't show
> security being enabled, but you do have it enabled, correct? Otherwise QOP
> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
> changing the QOP? Since your defaultFS is just "master", are you using HA?
>
> It's a bit concerning that you aren't consistently receiving the
> mismatch error. Is the client looping on retries and then you get timeouts
> after 5 attempts? If yes, we've got a major bug. 5 is the default number
> of RPC readers which handle SASL auth which means the protection mismatch
> is killing off the reader threads and rendering the NN unusable. This
> shouldn't be possible, but what does your NN log show?
>
> Daryn
>
> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>
>
>
> Hello,
>
> I am facing a problem in using Hadoop RPC encryption while transfer
> feature in hadoop 2.2.0. I have 3 node cluster
>
>
> Service running in node 1 (master)
> Resource manager
> Namenode
> DataNode
> SecondaryNamenode
>
> Service running in slaves ( node 2 & 3)
> NodeManager
>
>
>
> I am trying to make data transfer between master and slave secure. For
> that, I wanted to use data encryption over wire (RPC encryption) feature of
> hadoop 2.2.0
>
> When I ran the code, I get the below exception
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read.
>
>
> In another run, I saw in log the following error
>
> No common protection layer between server and client
>
> Not sure whether my configuration is inline with what I want to achieve.
>
> Can someone give me some hint on where I am going wrong?
>
> By the way, I have the below configuration setting in all of these nodes
>
> Core-site.xml
>
> <configuration>
>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://master:8020</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp</value>
> </property>
> <!--
> <property>
> <name>hadoop.rpc.protection</name>
> <value>privacy</value>
> </property>
> -->
> <property>
> <name>io.file.buffer.size</name>
> <value>131072</value>
> </property>
>
> </configuration>
>
> Hdfs-site.xml
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/app/hadoop/dfs-2.2.0/name</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/app/hadoop/dfs-2.2.0/data</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer.algorithm</name>
> <value>rc4</value>
> </property>
>
> <property>
> <name>dfs.block.access.token.enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Mapred-site.xml
>
> <configuration>
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <!--
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>master:8032</value>
> </property>
> -->
> <property>
> <name>mapreduce.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.map.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.reduce.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.map.java.opts</name>
> <value>-Xmx1024m</value>
> </property>
>
> </configuration>
>
>
> Yarn-site.xml
>
> <configuration>
>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>master</value>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
> </configuration>
>
>
>
> With thanks and regards
> Rab
>
>
>
>
>
>
Re: Problem with RPC encryption over wire
Posted by rab ra <ra...@gmail.com>.
Thank for the response.
I have removed rpc.protection parameter in all of my configuration and now
I am getting an error as below:-
Any Hint on whats going on here
13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
attempt_1384339616944_0002_m_000026_0, Status : FAILED
Container launch failed for container_1384339616944_0002_01_000029 :
java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)
at org.apache.hadoop.ipc.Client.call(Client.java:1351)
at org.apache.hadoop.ipc.Client.call(Client.java:1300)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy30.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)
13/11/14 10:10:50 INFO mapreduce.Job: map 72% reduce 24%
13/11/14 10:19:13 INFO mapreduce.Job: map 76% reduce 24%
13/11/14 10:19:55 INFO mapreduce.Job: map 78% reduce 24%
13/11/14 10:20:10 INFO mapreduce.Job: map 80% reduce 24%
13/11/14 10:20:29 INFO mapreduce.Job: map 81% reduce 24%
13/11/14 10:21:50 INFO mapreduce.Job: map 82% reduce 24%
13/11/14 10:22:01 INFO mapreduce.Job: map 83% reduce 24%
13/11/14 10:22:10 INFO mapreduce.Job: map 84% reduce 24%
13/11/14 10:22:13 INFO mapreduce.Job: map 85% reduce 24%
13/11/14 10:22:16 INFO mapreduce.Job: map 86% reduce 24%
13/11/14 10:22:27 INFO mapreduce.Job: map 86% reduce 28%
13/11/14 10:22:30 INFO mapreduce.Job: map 86% reduce 29%
13/11/14 10:28:30 INFO mapreduce.Job: map 88% reduce 29%
13/11/14 10:28:35 INFO mapreduce.Job: map 90% reduce 29%
13/11/14 10:28:37 INFO mapreduce.Job: map 97% reduce 29%
13/11/14 10:28:43 INFO mapreduce.Job: map 98% reduce 29%
13/11/14 10:28:45 INFO mapreduce.Job: map 99% reduce 29%
13/11/14 10:28:48 INFO mapreduce.Job: map 100% reduce 29%
13/11/14 10:28:55 INFO mapreduce.Job: map 100% reduce 33%
13/11/14 10:29:06 INFO mapreduce.Job: map 100% reduce 67%
13/11/14 10:29:27 INFO mapreduce.Job: map 100% reduce 100%
13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
successfully
13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server
13/11/14 10:29:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
[Error] Exception:
java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)
at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)
at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On 13 Nov 2013 20:11, "Daryn Sharp" <da...@yahoo-inc.com> wrote:
> "No common protection layer between server and client " likely means the
> host for job submission does not have hadoop.rpc.protection=privacy. In
> order for QOP to work, all client hosts (DN & others used to access the
> cluster) must have an identical setting.
>
> A few quick questions: I'm assuming you mis-posted your configs and the
> protection setting isn't really commented out? Your configs don't show
> security being enabled, but you do have it enabled, correct? Otherwise QOP
> shouldn't apply. Perhaps a bit obvious, but did you restart your NN after
> changing the QOP? Since your defaultFS is just "master", are you using HA?
>
> It's a bit concerning that you aren't consistently receiving the
> mismatch error. Is the client looping on retries and then you get timeouts
> after 5 attempts? If yes, we've got a major bug. 5 is the default number
> of RPC readers which handle SASL auth which means the protection mismatch
> is killing off the reader threads and rendering the NN unusable. This
> shouldn't be possible, but what does your NN log show?
>
> Daryn
>
> On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com> wrote:
>
>
>
> Hello,
>
> I am facing a problem in using Hadoop RPC encryption while transfer
> feature in hadoop 2.2.0. I have 3 node cluster
>
>
> Service running in node 1 (master)
> Resource manager
> Namenode
> DataNode
> SecondaryNamenode
>
> Service running in slaves ( node 2 & 3)
> NodeManager
>
>
>
> I am trying to make data transfer between master and slave secure. For
> that, I wanted to use data encryption over wire (RPC encryption) feature of
> hadoop 2.2.0
>
> When I ran the code, I get the below exception
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read.
>
>
> In another run, I saw in log the following error
>
> No common protection layer between server and client
>
> Not sure whether my configuration is inline with what I want to achieve.
>
> Can someone give me some hint on where I am going wrong?
>
> By the way, I have the below configuration setting in all of these nodes
>
> Core-site.xml
>
> <configuration>
>
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://master:8020</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/tmp</value>
> </property>
> <!--
> <property>
> <name>hadoop.rpc.protection</name>
> <value>privacy</value>
> </property>
> -->
> <property>
> <name>io.file.buffer.size</name>
> <value>131072</value>
> </property>
>
> </configuration>
>
> Hdfs-site.xml
> <configuration>
>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
>
> <property>
> <name>dfs.name.dir</name>
> <value>/app/hadoop/dfs-2.2.0/name</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/app/hadoop/dfs-2.2.0/data</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.encrypt.data.transfer.algorithm</name>
> <value>rc4</value>
> </property>
>
> <property>
> <name>dfs.block.access.token.enable</name>
> <value>true</value>
> </property>
>
> </configuration>
>
> Mapred-site.xml
>
> <configuration>
>
> <property>
> <name>mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <!--
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>master:8032</value>
> </property>
> -->
> <property>
> <name>mapreduce.tasktracker.map.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
> <value>1</value>
> </property>
>
> <property>
> <name>mapreduce.map.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.reduce.speculative</name>
> <value>false</value>
> </property>
>
> <property>
> <name>mapreduce.map.java.opts</name>
> <value>-Xmx1024m</value>
> </property>
>
> </configuration>
>
>
> Yarn-site.xml
>
> <configuration>
>
> <property>
> <name>yarn.resourcemanager.hostname</name>
> <value>master</value>
> </property>
>
> <property>
> <name>yarn.log-aggregation-enable</name>
> <value>true</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle</value>
> </property>
>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
>
> </configuration>
>
>
>
> With thanks and regards
> Rab
>
>
>
>
>
>
Re: Problem with RPC encryption over wire
Posted by Daryn Sharp <da...@yahoo-inc.com>.
"No common protection layer between server and client " likely means the host for job submission does not have hadoop.rpc.protection=privacy. In order for QOP to work, all client hosts (DN & others used to access the cluster) must have an identical setting.
A few quick questions: I'm assuming you mis-posted your configs and the protection setting isn't really commented out? Your configs don't show security being enabled, but you do have it enabled, correct? Otherwise QOP shouldn't apply. Perhaps a bit obvious, but did you restart your NN after changing the QOP? Since your defaultFS is just "master", are you using HA?
It's a bit concerning that you aren't consistently receiving the mismatch error. Is the client looping on retries and then you get timeouts after 5 attempts? If yes, we've got a major bug. 5 is the default number of RPC readers which handle SASL auth which means the protection mismatch is killing off the reader threads and rendering the NN unusable. This shouldn't be possible, but what does your NN log show?
Daryn
On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com>> wrote:
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 & 3)
NodeManager
I am trying to make data transfer between master and slave secure. For that, I wanted to use data encryption over wire (RPC encryption) feature of hadoop 2.2.0
When I ran the code, I get the below exception
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.
In another run, I saw in log the following error
No common protection layer between server and client
Not sure whether my configuration is inline with what I want to achieve.
Can someone give me some hint on where I am going wrong?
By the way, I have the below configuration setting in all of these nodes
Core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp</value>
</property>
<!--
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
-->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/dfs-2.2.0/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/dfs-2.2.0/data</value>
</property>
<property>
<name>dfs.encrypt.data.transfer</name>
<value>true</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>rc4</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name<http://mapreduce.framework.name/></name>
<value>yarn</value>
</property>
<!--
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8032</value>
</property>
-->
<property>
<name>mapreduce.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024m</value>
</property>
</configuration>
Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
With thanks and regards
Rab
Re: Problem with RPC encryption over wire
Posted by Daryn Sharp <da...@yahoo-inc.com>.
"No common protection layer between server and client " likely means the host for job submission does not have hadoop.rpc.protection=privacy. In order for QOP to work, all client hosts (DN & others used to access the cluster) must have an identical setting.
A few quick questions: I'm assuming you mis-posted your configs and the protection setting isn't really commented out? Your configs don't show security being enabled, but you do have it enabled, correct? Otherwise QOP shouldn't apply. Perhaps a bit obvious, but did you restart your NN after changing the QOP? Since your defaultFS is just "master", are you using HA?
It's a bit concerning that you aren't consistently receiving the mismatch error. Is the client looping on retries and then you get timeouts after 5 attempts? If yes, we've got a major bug. 5 is the default number of RPC readers which handle SASL auth which means the protection mismatch is killing off the reader threads and rendering the NN unusable. This shouldn't be possible, but what does your NN log show?
Daryn
On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com>> wrote:
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 & 3)
NodeManager
I am trying to make data transfer between master and slave secure. For that, I wanted to use data encryption over wire (RPC encryption) feature of hadoop 2.2.0
When I ran the code, I get the below exception
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.
In another run, I saw in log the following error
No common protection layer between server and client
Not sure whether my configuration is inline with what I want to achieve.
Can someone give me some hint on where I am going wrong?
By the way, I have the below configuration setting in all of these nodes
Core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp</value>
</property>
<!--
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
-->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/dfs-2.2.0/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/dfs-2.2.0/data</value>
</property>
<property>
<name>dfs.encrypt.data.transfer</name>
<value>true</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>rc4</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name<http://mapreduce.framework.name/></name>
<value>yarn</value>
</property>
<!--
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8032</value>
</property>
-->
<property>
<name>mapreduce.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024m</value>
</property>
</configuration>
Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
With thanks and regards
Rab
Re: Problem with RPC encryption over wire
Posted by Daryn Sharp <da...@yahoo-inc.com>.
"No common protection layer between server and client " likely means the host for job submission does not have hadoop.rpc.protection=privacy. In order for QOP to work, all client hosts (DN & others used to access the cluster) must have an identical setting.
A few quick questions: I'm assuming you mis-posted your configs and the protection setting isn't really commented out? Your configs don't show security being enabled, but you do have it enabled, correct? Otherwise QOP shouldn't apply. Perhaps a bit obvious, but did you restart your NN after changing the QOP? Since your defaultFS is just "master", are you using HA?
It's a bit concerning that you aren't consistently receiving the mismatch error. Is the client looping on retries and then you get timeouts after 5 attempts? If yes, we've got a major bug. 5 is the default number of RPC readers which handle SASL auth which means the protection mismatch is killing off the reader threads and rendering the NN unusable. This shouldn't be possible, but what does your NN log show?
Daryn
On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com>> wrote:
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 & 3)
NodeManager
I am trying to make data transfer between master and slave secure. For that, I wanted to use data encryption over wire (RPC encryption) feature of hadoop 2.2.0
When I ran the code, I get the below exception
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.
In another run, I saw in log the following error
No common protection layer between server and client
Not sure whether my configuration is inline with what I want to achieve.
Can someone give me some hint on where I am going wrong?
By the way, I have the below configuration setting in all of these nodes
Core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp</value>
</property>
<!--
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
-->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/dfs-2.2.0/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/dfs-2.2.0/data</value>
</property>
<property>
<name>dfs.encrypt.data.transfer</name>
<value>true</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>rc4</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name<http://mapreduce.framework.name/></name>
<value>yarn</value>
</property>
<!--
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8032</value>
</property>
-->
<property>
<name>mapreduce.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024m</value>
</property>
</configuration>
Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
With thanks and regards
Rab
Re: Problem with RPC encryption over wire
Posted by Daryn Sharp <da...@yahoo-inc.com>.
"No common protection layer between server and client " likely means the host for job submission does not have hadoop.rpc.protection=privacy. In order for QOP to work, all client hosts (DN & others used to access the cluster) must have an identical setting.
A few quick questions: I'm assuming you mis-posted your configs and the protection setting isn't really commented out? Your configs don't show security being enabled, but you do have it enabled, correct? Otherwise QOP shouldn't apply. Perhaps a bit obvious, but did you restart your NN after changing the QOP? Since your defaultFS is just "master", are you using HA?
It's a bit concerning that you aren't consistently receiving the mismatch error. Is the client looping on retries and then you get timeouts after 5 attempts? If yes, we've got a major bug. 5 is the default number of RPC readers which handle SASL auth which means the protection mismatch is killing off the reader threads and rendering the NN unusable. This shouldn't be possible, but what does your NN log show?
Daryn
On Nov 13, 2013, at 6:05 AM, rab ra <ra...@gmail.com>> wrote:
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 & 3)
NodeManager
I am trying to make data transfer between master and slave secure. For that, I wanted to use data encryption over wire (RPC encryption) feature of hadoop 2.2.0
When I ran the code, I get the below exception
Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read.
In another run, I saw in log the following error
No common protection layer between server and client
Not sure whether my configuration is inline with what I want to achieve.
Can someone give me some hint on where I am going wrong?
By the way, I have the below configuration setting in all of these nodes
Core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp</value>
</property>
<!--
<property>
<name>hadoop.rpc.protection</name>
<value>privacy</value>
</property>
-->
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/app/hadoop/dfs-2.2.0/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/app/hadoop/dfs-2.2.0/data</value>
</property>
<property>
<name>dfs.encrypt.data.transfer</name>
<value>true</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>rc4</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
</configuration>
Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name<http://mapreduce.framework.name/></name>
<value>yarn</value>
</property>
<!--
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8032</value>
</property>
-->
<property>
<name>mapreduce.tasktracker.map.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
</property>
<property>
<name>mapreduce.map.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.reduce.speculative</name>
<value>false</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx1024m</value>
</property>
</configuration>
Yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
With thanks and regards
Rab