You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Weiyi Kong (Jira)" <ji...@apache.org> on 2022/01/28 12:44:00 UTC

[jira] [Created] (SPARK-38056) Structured streaming not working in history server when using LevelDB

Weiyi Kong created SPARK-38056:
----------------------------------

             Summary: Structured streaming not working in history server when using LevelDB
                 Key: SPARK-38056
                 URL: https://issues.apache.org/jira/browse/SPARK-38056
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming, Web UI
    Affects Versions: 3.2.0, 3.1.2
            Reporter: Weiyi Kong
         Attachments: local-1643373518829

In [SPARK-31953|https://github.com/apache/spark/commit/4f9667035886a67e6c9a4e8fad2efa390e87ca68], structured streaming support is added to history server. However this does not work when spark.history.store.path is set to save app info using LevelDB.

This is because one of the keys of StreamingQueryData, runId,  is UUID type, which is not supported by LevelDB. When replaying event log file in history server, StreamingQueryStatusListener will throw an exception when writing info to the store, saying "java.lang.IllegalArgumentException: Type java.util.UUID not allowed as key.".

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org