You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2017/08/30 04:12:01 UTC
[jira] [Commented] (SPARK-21349) Make TASK_SIZE_TO_WARN_KB
configurable
[ https://issues.apache.org/jira/browse/SPARK-21349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16146569#comment-16146569 ]
Dongjoon Hyun commented on SPARK-21349:
---------------------------------------
Hi, [~jiangxb] and all. I hit this issue again in another situation today. So, I want to share the sample case.
{code}
scala> val data = (1 to (24*365*3)).map(i => (i, s"$i", i % 2 == 0)).toDF("col1", "part1", "part2")
17/08/29 21:07:49 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
data: org.apache.spark.sql.DataFrame = [col1: int, part1: string ... 1 more field]
scala> data.write.format("parquet").partitionBy("part1", "part2").mode("overwrite").saveAsTable("t")
17/08/29 21:08:04 WARN TaskSetManager: Stage 0 contains a task of very large size (190 KB). The maximum recommended task size is 100 KB.
17/08/29 21:09:34 WARN TaskSetManager: Stage 2 contains a task of very large size (233 KB). The maximum recommended task size is 100 KB.
scala> spark.version
res1: String = 2.3.0-SNAPSHOT
{code}
> Make TASK_SIZE_TO_WARN_KB configurable
> --------------------------------------
>
> Key: SPARK-21349
> URL: https://issues.apache.org/jira/browse/SPARK-21349
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.6.3, 2.2.0
> Reporter: Dongjoon Hyun
> Priority: Minor
>
> Since Spark 1.1.0, Spark emits warning when task size exceeds a threshold, SPARK-2185. Although this is just a warning message, this issue tries to make `TASK_SIZE_TO_WARN_KB` into a normal Spark configuration for advanced users.
> According to the Jenkins log, we also have 123 warnings even in our unit test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org