You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Vadim Chekan <ko...@gmail.com> on 2013/09/18 22:01:18 UTC
spark.cleaner.ttl default
Hi all,
I'm trying to load streaming context from a checkpoint: "val ctx = new
StreamingContext("\\Temp\\spark-checkpoint")" and I'm getting:
"Spark Streaming cannot be used without setting spark.cleaner.ttl; set this
property before creating a SparkContext (use SPARK_JAVA_OPTS for the shell)"
Apparently MetadataCleaner delay is set to default 3600 when streaming
context is created as "new StreamingContext("local", "cpc-stub",
Seconds(10))" but not when "new
StreamingContext("\\Temp\\spark-checkpoint")"
https://github.com/mesos/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L546
Is there a reason for spark.cleaner.ttl to be set to a default value in one
constructor but not in another? If it is just an omission, is github's
mesos/spark still preferred way to send pull request?
Vadim.
--
>From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is
explicitly specified
Re: spark.cleaner.ttl default
Posted by Matei Zaharia <ma...@gmail.com>.
Good catch, we should probably just load it from the checkpoint in the latter case.
Please send pull requests to apache/incubator-spark instead -- there are some new docs on it here: https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
Matei
On Sep 18, 2013, at 1:01 PM, Vadim Chekan <ko...@gmail.com> wrote:
> Hi all,
>
> I'm trying to load streaming context from a checkpoint: "val ctx = new
> StreamingContext("\\Temp\\spark-checkpoint")" and I'm getting:
> "Spark Streaming cannot be used without setting spark.cleaner.ttl; set this
> property before creating a SparkContext (use SPARK_JAVA_OPTS for the shell)"
>
> Apparently MetadataCleaner delay is set to default 3600 when streaming
> context is created as "new StreamingContext("local", "cpc-stub",
> Seconds(10))" but not when "new
> StreamingContext("\\Temp\\spark-checkpoint")"
>
> https://github.com/mesos/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L546
>
> Is there a reason for spark.cleaner.ttl to be set to a default value in one
> constructor but not in another? If it is just an omission, is github's
> mesos/spark still preferred way to send pull request?
>
> Vadim.
>
> --
> From RFC 2631: In ASN.1, EXPLICIT tagging is implicit unless IMPLICIT is
> explicitly specified