You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2016/11/07 19:25:59 UTC
[jira] [Resolved] (YARN-5842) spark job getting failed with memory
not avail
[ https://issues.apache.org/jira/browse/YARN-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash resolved YARN-5842.
--------------------------------
Resolution: Invalid
Please email user@hadoop.apache.org for questions. JIRA is for reporting confirmed bugs.
> spark job getting failed with memory not avail
> ----------------------------------------------
>
> Key: YARN-5842
> URL: https://issues.apache.org/jira/browse/YARN-5842
> Project: Hadoop YARN
> Issue Type: Task
> Components: applications
> Environment: spark running in emr 4.3 with hadoop 2.7 and spark 1.6.0
> Reporter: Mohamed Kajamoideen
>
> > config <- spark_config()
> > config$`sparklyr.shell.driver-memory` <- "4G"
> > config$`sparklyr.shell.executor-memory` <- "4G"
> > sc <- spark_connect(master = "yarn-client", config = config)
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 27.0 failed 4 times, most recent failure: Lost task 0.3 in stage 27.0 (TID 1941, ip- .ec2.internal): org.apache.spark.SparkException: Values to assemble cannot be null.
> at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:154)
> at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:137)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
> at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:137)
> at org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:95)
> at org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:94)
> at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Sou
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org