You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2017/06/20 21:42:00 UTC
[jira] [Resolved] (SPARK-19277) YARN topology script configuration
needs to be localized by Spark
[ https://issues.apache.org/jira/browse/SPARK-19277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-19277.
------------------------------------
Resolution: Won't Fix
After thinking a lot about this I think this is the wrong approach. The more correct fix is a combination of SPARK-9825 and making sure the cluster's config has the appropriate entries marked as "final". It's a little more awkward to deploy, but with automation it should be ok, and avoids the ugly code in the PR I closed.
> YARN topology script configuration needs to be localized by Spark
> -----------------------------------------------------------------
>
> Key: SPARK-19277
> URL: https://issues.apache.org/jira/browse/SPARK-19277
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.1.0
> Reporter: Marcelo Vanzin
> Priority: Minor
>
> (This really affects multiple versions, not just 2.1.0.)
> YARN has this configuration, {{net.topology.script.file.name}}, that defines a script to be run to figure out the cluster topology (which hosts are on which racks, etc). That configuration is generally a hardcoded path that is not parameterized; so when Spark runs the driver or AM on a separate host, that path may not exist, and then your log will be spammed with errors like this:
> {noformat}
> java.io.IOException: Cannot run program "/path/to/script" (in directory "/container/working/dir"): error=2, No such file or directory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:548)
> at org.apache.hadoop.util.Shell.run(Shell.java:504)
> at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
> at org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
> at org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
> at org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
> at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:95)
> at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.resolveRacks(AMRMClientImpl.java:548)
> at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.addContainerRequest(AMRMClientImpl.java:410)
> at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$updateResourceRequests$4.apply(YarnAllocator.scala:283)
> at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$updateResourceRequests$4.apply(YarnAllocator.scala:281)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> at org.apache.spark.deploy.yarn.YarnAllocator.updateResourceRequests(YarnAllocator.scala:281)
> at org.apache.spark.deploy.yarn.YarnAllocator.allocateResources(YarnAllocator.scala:220)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$1.run(ApplicationMaster.scala:368)
> Caused by: java.io.IOException: error=2, No such file or directory
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
> {noformat}
> Normally that error doesn't cause issues; at worst, task localization will be off because rack information is not available. But it's noisy, and if it happens enough, it may cause the YarnAllocator to slow down and cause other issues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org