You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2011/09/21 00:34:08 UTC
[jira] [Created] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
YARN Protobuf RPC Failures in RM
--------------------------------
Key: MAPREDUCE-3053
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.24.0
Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1173401
Node Kind: directory
Schedule: normal
Last Changed Author: vinodkv
Last Changed Rev: 1173130
Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
Reporter: Chris Riccomini
When I try to register my ApplicationMaster with YARN's RM, it fails.
In my ApplicationMaster's logs:
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
at $Proxy6.registerApplicationMaster(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
... 3 more
Caused by: java.lang.NullPointerException: java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
at org.apache.hadoop.ipc.Client.call(Client.java:1084)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
... 5 more
In the ResourceManager's logs:
2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
My registration code:
val appId = args(0).toInt
val attemptId = args(1).toInt
val timestamp = args(2).toLong
// these are our application master's parameters
val streamerClass = args(3)
val tasks = args(4).toInt
// TODO log params here
// start the application master helper
val conf = new Configuration
val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
.registerWithResourceManager
.....
val rpc = YarnRPC.create(conf)
val appId = Records.newRecord(classOf[ApplicationId])
val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
var requestId = 0
appId.setClusterTimestamp(lTimestamp)
appId.setId(iAppId)
appAttemptId.setApplicationId(appId)
appAttemptId.setAttemptId(iAppAttemptId)
def registerWithResourceManager(): ApplicationMasterHelper = {
val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
req.setApplicationAttemptId(appAttemptId)
// TODO not sure why these are blank- This is how spark does it
req.setHost("")
req.setRpcPort(1)
req.setTrackingUrl("")
resourceManager.registerApplicationMaster(req)
this
}
My params are receiving the proper app/attempt/cluster timestamps:
app - 1
attempt - 1
timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3053:
-----------------------------------------------
Attachment: MAPREDUCE-3053-20110922.txt
Attached simple patch for better message and a test to verify the same.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114299#comment-13114299 ]
Mahadev konar commented on MAPREDUCE-3053:
------------------------------------------
This is failing the tests:
{code}
Failed tests:
testUnknownCall(org.apache.hadoop.yarn.TestRPC): null expected:<...icationId called on []org.apache.hadoop.ya...> but was:<...icationId called on [interface ]org.apache.hadoop.ya...>
{code}
Not sure why hudson +1'ed it.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114202#comment-13114202 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #969 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/969/])
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf RPCs. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175357
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114198#comment-13114198 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1029 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1029/])
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf RPCs. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175357
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114251#comment-13114251 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #811 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/811/])
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf RPCs. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175357
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Riccomini resolved MAPREDUCE-3053.
----------------------------------------
Resolution: Fixed
Fixed with:
val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_SCHEDULER_ADDRESS, YarnConfiguration.DEFAULT_RM_SCHEDULER_ADDRESS))
See comment as to why.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114244#comment-13114244 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #841 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/841/])
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf RPCs. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175357
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109046#comment-13109046 ]
Mahadev konar commented on MAPREDUCE-3053:
------------------------------------------
Co incidentally, Hitesh [~hitesh] just ran into this :). You need to use YarnConfiguration.RM_SCHEDULER_ADDRESS to contact the RM from the AM. You are using YarnConfiguration.RM_ADDRESS which is the address used by the client to talk to the RM.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3053:
-----------------------------------------------
Affects Version/s: (was: 0.24.0)
0.23.0
Fix Version/s: 0.23.0
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114228#comment-13114228 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #20 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/20/])
Merge -r 1175356:1175357 from trunk to branch-0.23 to fix MAPREDUCE-3053.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175359
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3053:
-----------------------------------------------
Component/s: mrv2
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli reopened MAPREDUCE-3053:
------------------------------------------------
Assignee: Vinod Kumar Vavilapalli
I think it makes sense to fix the PB-RPC layer to check for wrong methods and throw a better error message.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112575#comment-13112575 ]
Hadoop QA commented on MAPREDUCE-3053:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496083/MAPREDUCE-3053-20110922.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/821//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/821//console
This message is automatically generated.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3053:
-----------------------------------------------
Status: Patch Available (was: Reopened)
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114197#comment-13114197 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #951 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/951/])
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf RPCs. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175357
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114340#comment-13114340 ]
Mahadev konar commented on MAPREDUCE-3053:
------------------------------------------
Fixed the test failure in MAPREDUCE-3054.
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13109045#comment-13109045 ]
Chris Riccomini commented on MAPREDUCE-3053:
--------------------------------------------
Sorry, I missed the call log on the RM C&P. It appears to fail on registerApplicationMaster (no surprise there).
Call: protocol=org.apache.hadoop.yarn.proto.AMRMProtocol$AMRMProtocolService$BlockingInterface, method=registerApplicationMaster
2011-09-20 15:36:52,506 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@7eb9132a from 127.0.0.1:49106, error:
java.lang.NullPointerException
at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.24.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3053) YARN Protobuf RPC Failures in
RM
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114239#comment-13114239 ]
Hudson commented on MAPREDUCE-3053:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #25 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/25/])
Merge -r 1175356:1175357 from trunk to branch-0.23 to fix MAPREDUCE-3053.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175359
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3053) YARN Protobuf RPC Failures in RM
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3053:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Vinod!
> YARN Protobuf RPC Failures in RM
> --------------------------------
>
> Key: MAPREDUCE-3053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3053
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Environment: URL: http://svn.apache.org/repos/asf/hadoop/common/trunk
> Repository Root: http://svn.apache.org/repos/asf
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
> Revision: 1173401
> Node Kind: directory
> Schedule: normal
> Last Changed Author: vinodkv
> Last Changed Rev: 1173130
> Last Changed Date: 2011-09-20 06:11:12 -0700 (Tue, 20 Sep 2011)
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3053-20110922.txt
>
>
> When I try to register my ApplicationMaster with YARN's RM, it fails.
> In my ApplicationMaster's logs:
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:108)
> at kafka.yarn.util.ApplicationMasterHelper.registerWithResourceManager(YarnHelper.scala:48)
> at kafka.yarn.ApplicationMaster$.main(ApplicationMaster.scala:32)
> at kafka.yarn.ApplicationMaster.main(ApplicationMaster.scala)
> Caused by: com.google.protobuf.ServiceException: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:130)
> at $Proxy6.registerApplicationMaster(Unknown Source)
> at org.apache.hadoop.yarn.api.impl.pb.client.AMRMProtocolPBClientImpl.registerApplicationMaster(AMRMProtocolPBClientImpl.java:101)
> ... 3 more
> Caused by: java.lang.NullPointerException: java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> at org.apache.hadoop.ipc.Client.call(Client.java:1084)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:127)
> ... 5 more
> In the ResourceManager's logs:
> 2011-09-20 15:11:20,973 INFO ipc.Server (Server.java:run(1497)) - IPC Server handler 2 on 8040, call: org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$ProtoSpecificRequestWritable@455dd32a from 127.0.0.1:33793, error:
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.proto.ClientRMProtocol$ClientRMProtocolService$2.getRequestPrototype(ClientRMProtocol.java:186)
> at org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:323)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1489)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1485)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1483)
> My registration code:
> val appId = args(0).toInt
> val attemptId = args(1).toInt
> val timestamp = args(2).toLong
> // these are our application master's parameters
> val streamerClass = args(3)
> val tasks = args(4).toInt
> // TODO log params here
> // start the application master helper
> val conf = new Configuration
> val applicationMasterHelper = new ApplicationMasterHelper(appId, attemptId, timestamp, conf)
> .registerWithResourceManager
> .....
> val rpc = YarnRPC.create(conf)
> val appId = Records.newRecord(classOf[ApplicationId])
> val appAttemptId = Records.newRecord(classOf[ApplicationAttemptId])
> val rmAddress = NetUtils.createSocketAddr(conf.get(YarnConfiguration.RM_ADDRESS, YarnConfiguration.DEFAULT_RM_ADDRESS))
> val resourceManager = rpc.getProxy(classOf[AMRMProtocol], rmAddress, conf).asInstanceOf[AMRMProtocol]
> var requestId = 0
> appId.setClusterTimestamp(lTimestamp)
> appId.setId(iAppId)
> appAttemptId.setApplicationId(appId)
> appAttemptId.setAttemptId(iAppAttemptId)
> def registerWithResourceManager(): ApplicationMasterHelper = {
> val req = Records.newRecord(classOf[RegisterApplicationMasterRequest])
> req.setApplicationAttemptId(appAttemptId)
> // TODO not sure why these are blank- This is how spark does it
> req.setHost("")
> req.setRpcPort(1)
> req.setTrackingUrl("")
> resourceManager.registerApplicationMaster(req)
> this
> }
> My params are receiving the proper app/attempt/cluster timestamps:
> app - 1
> attempt - 1
> timestamp - 1316556657998
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira