[jira] [Resolved] (SPARK-19316) Spark event logs are huge compared to 1.5.2

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-19316.
-------------------------------
    Resolution: Duplicate

Please search JIRA first

> Spark event logs are huge compared to 1.5.2
> -------------------------------------------
>
>                 Key: SPARK-19316
>                 URL: https://issues.apache.org/jira/browse/SPARK-19316
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.0.0
>            Reporter: Jisoo Kim
>
> I have a Spark application with many tasks (more than 40k). The event logs for such application used to be around 2g when I was using Spark 1.5.2 standalone cluster. Now that I am using Spark 2.0 with Mesos, the size of the event log of such application drastically increased from 2g to 60g with a similar number of tasks. This is affecting Spark History Server since it is having trouble reading such huge event log. I wonder the increase in a size of an event log is expected in Spark 2.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org