You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by akhandeshi <am...@gmail.com> on 2014/10/28 22:32:25 UTC
Spark-submt job "Killed"
I recently starting seeing this new problem where spark-submt is terminated
by "Killed" message but no error message indicating what happened. I have
enable logging on in spark configuration. has anyone seen this or know how
to troubleshoot?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submt-job-Killed-tp17560.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark-submt job "Killed"
Posted by akhandeshi <am...@gmail.com>.
I did have it as rdd.saveAsText("RDD");
and now I have it as:
Log.info("RDD
Counts"+rdd.persist(StorageLevel.MEMORY_AND_DISK_SER()).count());
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submt-job-Killed-tp17560p17598.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark-submt job "Killed"
Posted by "Ganelin, Ilya" <Il...@capitalone.com>.
Hi Ami - I suspect that your code is completing because you have nothing
to actually force resolution of your job. Spark executes lazily, so for
example, if you have a bunch of maps in sequence but nothing else, Spark
will not actually execute anything.
Try adding an RDD.count() on the last RDD that you generate with spark to
ensure that something is forcing execution at the end.
On 10/28/14, 5:32 PM, "akhandeshi" <am...@gmail.com> wrote:
>I recently starting seeing this new problem where spark-submt is
>terminated
>by "Killed" message but no error message indicating what happened. I have
>enable logging on in spark configuration. has anyone seen this or know
>how
>to troubleshoot?
>
>
>
>
>--
>View this message in context:
>http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submt-job-Killed
>-tp17560.html
>Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>For additional commands, e-mail: user-help@spark.apache.org
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org