You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2018/02/26 00:08:00 UTC
[jira] [Assigned] (SPARK-23515) JsonProtocol.sparkEventToJson can
OOM when jsonifying an event
[ https://issues.apache.org/jira/browse/SPARK-23515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-23515:
------------------------------------
Assignee: Apache Spark (was: Burak Yavuz)
> JsonProtocol.sparkEventToJson can OOM when jsonifying an event
> --------------------------------------------------------------
>
> Key: SPARK-23515
> URL: https://issues.apache.org/jira/browse/SPARK-23515
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.1
> Reporter: Burak Yavuz
> Assignee: Apache Spark
> Priority: Major
>
> {code}
> def sparkEventToJson(event: SparkListenerEvent)
> {code}
> has a fallback method which creates a JSON object by turning an unrecognized event to Json and then parsing it again. This method materializes the whole string to parse the json record, which is unnecessary and can cause OOMs as seen in the stacktrace below:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3664)
> at java.lang.String.<init>(String.java:207)
> at java.lang.StringBuilder.toString(StringBuilder.java:407)
> at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:356)
> at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:235)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:20)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
> at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
> at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
> at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
> at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:20)
> at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:50)
> at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:103){code}
>
> We should just use the stream parsing to avoid such OOMs.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org