[jira] [Created] (SPARK-43973) Structured Streaming UI should display failed queries correctly

["Kris Mok (Jira)" <ji...@apache.org>]

Kris Mok created SPARK-43973:
--------------------------------

             Summary: Structured Streaming UI should display failed queries correctly
                 Key: SPARK-43973
                 URL: https://issues.apache.org/jira/browse/SPARK-43973
             Project: Spark
          Issue Type: Bug
          Components: Web UI
    Affects Versions: 3.4.0, 3.3.0, 3.2.0, 3.1.0
            Reporter: Kris Mok


The Structured Streaming UI is designed to be able to show a query's status (active/finished/failed) and if failed, the error message.
Due to a bug in the implementation, the error message in {{QueryTerminatedEvent}} isn't being tracked by the UI data, so in turn the UI always shows failed queries as "finished".

Example:
{code:scala}
implicit val ctx = spark.sqlContext
import org.apache.spark.sql.execution.streaming.MemoryStream

spark.conf.set("spark.sql.ansi.enabled", "true")

val inputData = MemoryStream[(Int, Int)]

val df = inputData.toDF().selectExpr("_1 / _2 as a")

inputData.addData((1, 2), (3, 4), (5, 6), (7, 0))
val testQuery = df.writeStream.format("memory").queryName("kristest").outputMode("append").start
testQuery.processAllAvailable()
{code}

Here we intentionally fail a query, but the Spark UI's Structured Streaming tab would show this as "FINISHED" without any errors, which is wrong.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org