You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/04/30 08:06:30 UTC
spark git commit: [SPARK-20521][DOC][CORE] The default of
'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md
Repository: spark
Updated Branches:
refs/heads/master d228cd0b0 -> 4d99b95ad
[SPARK-20521][DOC][CORE] The default of 'spark.worker.cleanup.appDataTtl' should be 604800 in spark-standalone.md
## What changes were proposed in this pull request?
Currently, our project needs to be set to clean up the worker directory cleanup cycle is three days.
When I follow http://spark.apache.org/docs/latest/spark-standalone.html, configure the 'spark.worker.cleanup.appDataTtl' parameter, I configured to 3 * 24 * 3600.
When I start the spark service, the startup fails, and the worker log displays the error log as follows:
2017-04-28 15:02:03,306 INFO Utils: Successfully started service 'sparkWorker' on port 48728.
Exception in thread "main" java.lang.NumberFormatException: For input string: "3 * 24 * 3600"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:430)
at java.lang.Long.parseLong(Long.java:483)
at scala.collection.immutable.StringLike$class.toLong(StringLike.scala:276)
at scala.collection.immutable.StringOps.toLong(StringOps.scala:29)
at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
at org.apache.spark.SparkConf$$anonfun$getLong$2.apply(SparkConf.scala:380)
at scala.Option.map(Option.scala:146)
at org.apache.spark.SparkConf.getLong(SparkConf.scala:380)
at org.apache.spark.deploy.worker.Worker.<init>(Worker.scala:100)
at org.apache.spark.deploy.worker.Worker$.startRpcEnvAndEndpoint(Worker.scala:730)
at org.apache.spark.deploy.worker.Worker$.main(Worker.scala:709)
at org.apache.spark.deploy.worker.Worker.main(Worker.scala)
**Because we put 7 * 24 * 3600 as a string, forced to convert to the dragon type, will lead to problems in the program.**
**So I think the default value of the current configuration should be a specific long value, rather than 7 * 24 * 3600,should be 604800. Because it would mislead users for similar configurations, resulting in spark start failure.**
## How was this patch tested?
manual tests
Please review http://spark.apache.org/contributing.html before opening a pull request.
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Author: guoxiaolong <gu...@zte.com.cn>
Author: guoxiaolongzte <gu...@zte.com.cn>
Closes #17798 from guoxiaolongzte/SPARK-20521.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4d99b95a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4d99b95a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4d99b95a
Branch: refs/heads/master
Commit: 4d99b95ad0d0c7ef909c8e492ec45e94cf0189b4
Parents: d228cd0
Author: \u90ed\u5c0f\u9f99 10207633 <gu...@zte.com.cn>
Authored: Sun Apr 30 09:06:25 2017 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Sun Apr 30 09:06:25 2017 +0100
----------------------------------------------------------------------
docs/spark-standalone.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/4d99b95a/docs/spark-standalone.md
----------------------------------------------------------------------
diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md
index 1c0b60f..34ced9e 100644
--- a/docs/spark-standalone.md
+++ b/docs/spark-standalone.md
@@ -242,7 +242,7 @@ SPARK_WORKER_OPTS supports the following system properties:
</tr>
<tr>
<td><code>spark.worker.cleanup.appDataTtl</code></td>
- <td>7 * 24 * 3600 (7 days)</td>
+ <td>604800 (7 days, 7 * 24 * 3600)</td>
<td>
The number of seconds to retain application work directories on each worker. This is a Time To Live
and should depend on the amount of available disk space you have. Application logs and jars are
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org