[jira] [Created] (SPARK-32076) Structured Streaming application continuity when encountering streaming query task level error

["Jim Huang (Jira)" <ji...@apache.org>]

Jim Huang created SPARK-32076:
---------------------------------

             Summary: Structured Streaming application continuity when encountering streaming query task level error
                 Key: SPARK-32076
                 URL: https://issues.apache.org/jira/browse/SPARK-32076
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.4.5
         Environment: Apache Spark 2.4.5 Scala 2.11 without Hadoop

Hadoop 2.7.3 - YARN cluster

delta-core_ 2.11:0.6.1
            Reporter: Jim Huang


From the Spark Structured Streaming application continuity perspective, the thread that ran this task was terminated with ERROR SPARK-31995 but to YARN it is still an active running job even though this instance of the Spark Structured Streaming job is no longer making any further processing.  If the monitoring of the Spark Structured Streaming job is done only from the YARN job perspective, it may provide a false status.  In this situation, should the Spark Structure Streaming application fail hard and completely (fail by Spark framework or Application exception handling)?  Or should the developer investigate and develop some ideal monitoring implementation that has the right level of specificity to detect Spark Structured Streaming *task* level failures?  Any references on these topics are much appreciated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org