You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Francesco Scipioni (Jira)" <ji...@apache.org> on 2022/10/13 13:28:00 UTC
[jira] [Commented] (HADOOP-18495) Configuration Resolver not working on env.vars through namenode/conf or spark
[ https://issues.apache.org/jira/browse/HADOOP-18495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617053#comment-17617053 ]
Francesco Scipioni commented on HADOOP-18495:
---------------------------------------------
Application application_1665665258262_0001 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1665665258262_0001_000001 exited with exitCode: 10
Failing this attempt.Diagnostics: [2022-10-13 12:53:12.477]Exception from container-launch.
Container id: container_1665665258262_0001_01_000001
Exit code: 10
[2022-10-13 12:53:12.513]Container exited with a non-zero exit code 10. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
oxy(RMProxy.java:103)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.serviceStart(AMRMClientImpl.java:193)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.spark.deploy.yarn.YarnRMClient.register(YarnRMClient.scala:63)
at org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:440)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:551)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:277)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:926)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:925)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:925)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:957)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
2022-10-13 12:53:11,689 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.IllegalArgumentException: Does not contain a valid host:port authority: ${env.MASTER_HOSTNAME}:8030 (configuration property 'yarn.resourcemanager.scheduler.address')
at org.apache.hadoop.net.NetUtils.createURI(NetUtils.java:262)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:221)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:189)
at org.apache.hadoop.yarn.conf.YarnConfiguration.getSocketAddr(YarnConfiguration.java:4607)
at org.apache.hadoop.yarn.client.ClientRMProxy.getTokenService(ClientRMProxy.java:162)
at org.apache.hadoop.yarn.client.ClientRMProxy.getAMRMTokenService(ClientRMProxy.java:139)
at org.apache.hadoop.yarn.client.ClientRMProxy.setAMRMTokenService(ClientRMProxy.java:81)
at org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:100)
at org.apache.hadoop.yarn.client.DefaultNoHARMFailoverProxyProvider.init(DefaultNoHARMFailoverProxyProvider.java:63)
at org.apache.hadoop.yarn.client.RMProxy.createNonHaRMFailoverProxyProvider(RMProxy.java:172)
at org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:132)
at org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:103)
at org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:73)
at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.serviceStart(AMRMClientImpl.java:193)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.spark.deploy.yarn.YarnRMClient.register(YarnRMClient.scala:63)
at org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:440)
at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:551)
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:277)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:926)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:925)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:925)
at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:957)
at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
)
so you mean that i should set on yarn-site.xml
{code:xml}
<property>
<name>yarn.nodemanager.admin-env.MASTER_HOSTNAME</name>
<value>${env.MASTER_HOSTNAME}</value>
<description>To define environment variables individually, you can specify multiple properties of the form yarn.nodemanager.admin-env.VARNAME, where VARNAME is the name of the environment variable. This is the only way to add a variable when its value contains commas.</description>
</property>
{code}
> Configuration Resolver not working on env.vars through namenode/conf or spark
> -----------------------------------------------------------------------------
>
> Key: HADOOP-18495
> URL: https://issues.apache.org/jira/browse/HADOOP-18495
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
> Affects Versions: 3.3.4
> Environment: Hadoop 3.3.4 + Spark 3.3.0
> Reporter: Francesco Scipioni
> Priority: Major
>
> - Following [Configuration javadoc|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java]
> - Using property value as
> {code:xml}
> ${env.ENV_VARIABLE}
> {code}
> - Cluster Startup well, and property is correctly replaced
> Launching a Spark Job or accessing configuration throught http://namenode/conf property value is not replaced and stile provided as value ${env.ENV_VARIABLE}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org