You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shao Feng Shi (Jira)" <ji...@apache.org> on 2020/04/29 07:37:00 UTC
[jira] [Closed] (KYLIN-3726) KylinSession should load spark
properties from spark-defaults.conf
[ https://issues.apache.org/jira/browse/KYLIN-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shao Feng Shi closed KYLIN-3726.
--------------------------------
> KylinSession should load spark properties from spark-defaults.conf
> ------------------------------------------------------------------
>
> Key: KYLIN-3726
> URL: https://issues.apache.org/jira/browse/KYLIN-3726
> Project: Kylin
> Issue Type: Sub-task
> Components: Storage - Parquet
> Reporter: Shaohui Liu
> Assignee: Shaohui Liu
> Priority: Major
>
> When testing parquet storage, the spark session job failed to be submit for no JAVA_HOME in executor env. This config is set in the spark default property file: spark-defaults.conf.
>
> {code}
> 2018-12-18,15:13:15,466 ERROR org.apache.spark.deploy.yarn.YarnAllocator: Failed to launch executor 6 on container container_e823_1541646991414_1025309_01_000007 java.util.NoSuchElementException: key not found: JAVA_HOME at scala.collection.MapLike$class.default(MapLike.scala:228) at scala.collection.AbstractMap.default(Map.scala:59) at scala.collection.mutable.HashMap.apply(HashMap.scala:65) at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:286) at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3$$anonfun$apply$3.apply(ExecutorRunnable.scala:275) at scala.Option.foreach(Option.scala:257) at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:275) at org.apache.spark.deploy.yarn.ExecutorRunnable$$anonfun$prepareEnvironment$3.apply(ExecutorRunnable.scala:274) at scala.Option.foreach(Option.scala:257) at org.apache.spark.deploy.yarn.ExecutorRunnable.prepareEnvironment(ExecutorRunnable.scala:274) at org.apache.spark.deploy.yarn.ExecutorRunnable.startContainer(ExecutorRunnable.scala:92) at org.apache.spark.deploy.yarn.ExecutorRunnable.run(ExecutorRunnable.scala:69) at org.apache.spark.deploy.yarn.YarnAllocator$$anonfun$runAllocatedContainers$1$$anon$1.run(YarnAllocator.scala:556) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)