[GitHub] [spark] pengzhon-db commented on a diff in pull request #40785: [SPARK-42960] Add await_termination() and exception() API for Streaming Query

["pengzhon-db (via GitHub)" <gi...@apache.org>]

pengzhon-db commented on code in PR #40785:
URL: https://github.com/apache/spark/pull/40785#discussion_r1169111168


##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -298,6 +309,18 @@ message StreamingQueryCommandResult {
     // Logical and physical plans as string
     string result = 1;
   }
+
+  message ExceptionResult {
+    // Exception message as string
+    oneof exception_content {
+      bool no_exception = 1;

Review Comment:
   why need to have this? Can we just not set the exception field if there is no exception?



##########
connector/connect/common/src/main/protobuf/spark/connect/commands.proto:
##########
@@ -298,6 +306,16 @@ message StreamingQueryCommandResult {
     // Logical and physical plans as string
     string result = 1;
   }
+
+  message ExceptionResult {

Review Comment:
   I think we should try to be consistent with what current `exception()` returns, which is `return CapturedStreamingQueryException(msg, stackTrace, je.getCause())` ?



##########
python/pyspark/sql/connect/streaming/query.py:
##########
@@ -24,6 +25,9 @@
 from pyspark.sql.streaming.query import (
     StreamingQuery as PySparkStreamingQuery,
 )
+from pyspark.errors.exceptions.connect import (
+    StreamingQueryException as CapturedStreamingQueryException,

Review Comment:
   why need this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org