[jira] [Created] (SPARK-23515) JsonProtocol.sparkEventToJson can OOM when jsonifying an event

["Burak Yavuz (JIRA)" <ji...@apache.org>]

Burak Yavuz created SPARK-23515:
-----------------------------------

             Summary: JsonProtocol.sparkEventToJson can OOM when jsonifying an event
                 Key: SPARK-23515
                 URL: https://issues.apache.org/jira/browse/SPARK-23515
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.2.1
            Reporter: Burak Yavuz
            Assignee: Burak Yavuz


{code}

def sparkEventToJson(event: SparkListenerEvent)

{code}

has a fallback method which creates a JSON object by turning an unrecognized event to Json and then parsing it again. This method materializes the whole string to parse the json record, which is unnecessary and can cause OOMs as seen in the stacktrace below:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:356)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:235)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:20)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
at org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:20)
at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:50)
at org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:103){code}
 

We should just use the stream parsing to avoid such OOMs.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org