You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "heyihong (via GitHub)" <gi...@apache.org> on 2023/08/07 10:01:02 UTC

[GitHub] [spark] heyihong opened a new pull request, #42377: [SPARK-44622][SQL][CONNECT] Add GetErrorInfo RPC

heyihong opened a new pull request, #42377:
URL: https://github.com/apache/spark/pull/42377

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'core/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1687891266

   > I think this needs a signoff from either @grundprinzip or @hvanhovell
   
   Of course


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323060159


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage ...
   
   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 ...
   
   org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   
   java.lang.RuntimeException: Number 2 encountered!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330278768


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)

Review Comment:
   It's good to put a comment here that this error can only be fetched once, if a connection dies in the middle you cannot repeat.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330327877


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I changed to use optional 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330599584


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   Not your code, but are we seriously using a lazy val here to avoid one level of nesting? that seems excessive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329197325


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -3292,39 +3292,49 @@ def test_stop_session(self):
             self.spark.conf.get("some.conf")
         self._check_no_active_session_error(e.exception)
 
-    def test_error_stack_trace(self):
-        with self.sql_conf({"spark.sql.pyspark.jvmStacktrace.enabled": True}):

Review Comment:
   Doesn't this effectively change the API? What is the expectation when the user sets the SQL conf and moves from py4j to spark connect and back? Shouldn't we retain the option?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329325417


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   It is possible if a client with newer version connects to a server with older version or enrich error is disabled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329968772


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   This is in the client library and overall the client seems to lack of logging support at the moment. For improving the logging can we do it as a follow up ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309581203


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +44,40 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test(s"throw SparkException with large cause exception") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {

Review Comment:
   Connect is in connect/common... so you can use it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312528218


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2842,17 +2842,6 @@ object SQLConf {
       .checkValue(_ >= 0, "This value must be equal to or greater than 0.")
       .createWithDefaultString("48MB")
 
-  val PYSPARK_JVM_STACKTRACE_ENABLED =
-    buildConf("spark.sql.pyspark.jvmStacktrace.enabled")

Review Comment:
   This is a user facing conf right? I don't think we can just remove this outright.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329212299


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   IIRC the agreed upon behavior was that we keep everything as is, but we simply add an error ID that the client can use to fetch the full error stack trace. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329204308


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          isTruncated = false,
+          serverStackTraceEnabled =
+            sessionHolder.session.conf.get(Connect.CONNECT_SERVER_STACKTRACE_ENABLED),
+          pySparkJVMStackTraceEnabled =
+            sessionHolder.session.conf.get(SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),

Review Comment:
   For me as the reader it's hard to reason about that the difference between the two is?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330596618


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,7 +76,59 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false): FetchErrorDetailsResponse = {
+    val errors =
+      traverseCauses(st).take(MAX_ERROR_CHAIN_LENGTH).zipWithIndex.map { case (error, idx) =>
+        val builder = FetchErrorDetailsResponse.Error
+          .newBuilder()
+          .setMessage(error.getMessage)
+          .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+        if (error.getCause != null) {
+          builder.setCauseIdx(idx + 1)

Review Comment:
   Sorry for harping on here, two things:
   1. This breaks if reach `MAX_ERROR_CHAIN_LENGTH`, you will be point to an exception you cannot reach.
   2. In terms of control this is less than great because you assume that `traverseCauses` produces a certain order.  In this case an imperative loop might have been better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed (maybe some bugs on the server side, the exception is no longer cached), fallback to an abbreviated FetchErrorDetailsResponse is more consistent with the existing control flow. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control flow. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330279785


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I'm not sure if empty list is a good indicator to show that the error did not exist.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plans, they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions IMO... I can change it back...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330169939


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +46,13 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  // The cache that maps an error id to a throwable. The throwable in cache is independent to
+  // each other.
+  val errorIdToError =
+    SparkConnectService
+      .cacheBuilder(ERROR_CACHE_SIZE, ERROR_CACHE_TIMEOUT_SEC)

Review Comment:
   LRU



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330261417


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,

Review Comment:
   Also merge them to one



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309586551


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -40,7 +42,15 @@ import org.apache.spark.sql.connect.service.SparkConnectService
 import org.apache.spark.sql.internal.SQLConf
 
 private[connect] object ErrorUtils extends Logging {
-  private def allClasses(cl: Class[_]): Seq[Class[_]] = {
+  private[connect] def allThrowables(t: Throwable): List[Throwable] = {
+    if (t == null) {
+      List.empty
+    } else {
+      t :: allThrowables(t.getCause)

Review Comment:
   What about suppressed throwables?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310919492


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,38 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {
+      val session = spark
+      import session.implicits._
+
+      val checkForTwo = udf((value: Int) => {
+        if (value == 2) {
+          throw new RuntimeException("Number 2 encountered!")
+        }
+        value
+      })
+
+      val query = spark.readStream
+        .format("rate")
+        .option("rowsPerSecond", "1")
+        .load()
+        .select(checkForTwo($"value").as("checkedValue"))
+        .writeStream
+        .outputMode("append")
+        .format("console")
+        .start()
+
+      val exception = intercept[SparkException] {

Review Comment:
   StreamingQueryException requires some refactoring since spark and connect use different StreamingQueryException at the moment. Will do this in a follow up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310921357


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -40,7 +42,15 @@ import org.apache.spark.sql.connect.service.SparkConnectService
 import org.apache.spark.sql.internal.SQLConf
 
 private[connect] object ErrorUtils extends Logging {
-  private def allClasses(cl: Class[_]): Seq[Class[_]] = {
+  private[connect] def allThrowables(t: Throwable): List[Throwable] = {
+    if (t == null) {
+      List.empty
+    } else {
+      t :: allThrowables(t.getCause)

Review Comment:
   I took a look but didn't find a use case. Do we actually use suppresed throwables in Spark? If not, I think we can suppress them for now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309579814


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +183,38 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(status.getMessage, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      errorId: String,
+      classes: Array[String],

Review Comment:
   I think this name could be better. It is the class hierarchy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322934109


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +218,34 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      classHierarchy: Array[String],
+      message: String,
+      stackTraceOpt: Option[Array[StackTraceElement]],
+      causeOpt: Option[Error])
+
+  private def errorToThrowable(error: Error): Option[Throwable] = {

Review Comment:
   How does this interact with the error framework? Do we retain all the relevant fields?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1300892182


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +44,40 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test(s"throw SparkException with large cause exception") {

Review Comment:
   ```suggestion
     test("throw SparkException with large cause exception") {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309568029


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(
+        GetErrorInfoRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .addAllErrorIds(errors.map(_.errorId).toIterable.asJava)
+          .setUserContext(UserContext.newBuilder().setUserId(userId).build())
+          .build())
+
+      val exceptionInfos = errorInfoResponse.getExceptionInfosList.asScala

Review Comment:
   Make it a map? Instead of brute forcing things?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309573632


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {
+    // Unique identifier for this specific exception instance.
+    string error_id = 1;
+
+    // Fully qualified names of the exception class and its parental classes.

Review Comment:
   Nit parental -> parent



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309562944


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   @heyihong 3.5.0 is too late.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330178093


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   For example, if we want to include the stack trace of cause exceptions in the error messages (currently only the stacktrace of the root exception), there are more works for all non-jvm clients (and harder to change)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330657004


##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {
+
+  private val userId = "user1"
+
+  private val sessionId = UUID.randomUUID().toString
+
+  private def fetchErrorDetails(
+      userId: String,
+      sessionId: String,
+      errorId: String): FetchErrorDetailsResponse = {
+    val promise = Promise[FetchErrorDetailsResponse]
+    val handler =
+      new SparkConnectFetchErrorDetailsHandler(new FetchErrorDetailsResponseObserver(promise))
+    val context = proto.UserContext
+      .newBuilder()
+      .setUserId(userId)
+      .build()
+    val request = proto.FetchErrorDetailsRequest
+      .newBuilder()
+      .setUserContext(context)
+      .setSessionId(sessionId)
+      .setErrorId(errorId)
+      .build()
+    handler.handle(request)
+    ThreadUtils.awaitResult(promise.future, 5.seconds)
+  }
+
+  test("error chain is properly constructed") {
+    val testError =
+      new Exception("test1", new Exception("test2"))
+    val errorId = UUID.randomUUID().toString()
+
+    SparkConnectService
+      .getOrCreateIsolatedSession(userId, sessionId)
+      .errorIdToError
+      .put(errorId, testError)
+
+    val response = fetchErrorDetails(userId, sessionId, errorId)
+    assert(response.hasRootErrorIdx)
+    assert(response.getRootErrorIdx == 0)
+
+    assert(response.getErrorsCount == 2)
+    assert(response.getErrors(0).getMessage == "test1")
+    assert(response.getErrors(0).getErrorTypeHierarchy(0) == classOf[Exception].getName)

Review Comment:
   Exception, Throwable, Object



##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {
+
+  private val userId = "user1"
+
+  private val sessionId = UUID.randomUUID().toString
+
+  private def fetchErrorDetails(
+      userId: String,
+      sessionId: String,
+      errorId: String): FetchErrorDetailsResponse = {
+    val promise = Promise[FetchErrorDetailsResponse]
+    val handler =
+      new SparkConnectFetchErrorDetailsHandler(new FetchErrorDetailsResponseObserver(promise))
+    val context = proto.UserContext
+      .newBuilder()
+      .setUserId(userId)
+      .build()
+    val request = proto.FetchErrorDetailsRequest
+      .newBuilder()
+      .setUserContext(context)
+      .setSessionId(sessionId)
+      .setErrorId(errorId)
+      .build()
+    handler.handle(request)
+    ThreadUtils.awaitResult(promise.future, 5.seconds)
+  }
+
+  test("error chain is properly constructed") {
+    val testError =
+      new Exception("test1", new Exception("test2"))
+    val errorId = UUID.randomUUID().toString()
+
+    SparkConnectService
+      .getOrCreateIsolatedSession(userId, sessionId)
+      .errorIdToError
+      .put(errorId, testError)
+
+    val response = fetchErrorDetails(userId, sessionId, errorId)
+    assert(response.hasRootErrorIdx)
+    assert(response.getRootErrorIdx == 0)
+
+    assert(response.getErrorsCount == 2)
+    assert(response.getErrors(0).getMessage == "test1")
+    assert(response.getErrors(0).getErrorTypeHierarchy(0) == classOf[Exception].getName)

Review Comment:
   Added these checks into the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330327877


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I think it depends on the meaning of an empty list. But I changed to use optional 



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I think it depends on the meaning of an empty list. I changed to use optional 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed (maybe some bugs on the server side, the exception is no longer cached), fallback to an abbreviated FetchErrorDetailsResponse is more consistent with the existing control flow. Otherwise, we may have to throw a different Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed (maybe some bugs on the server side, the exception is no longer cached), fallback to an abbreviated FetchErrorDetailsResponse is more consistent with the existing control flow. Otherwise, we may have to throw a different Exception or fallback to an exception without causes (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330633746


##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {
+
+  private val userId = "user1"
+
+  private val sessionId = UUID.randomUUID().toString
+
+  private def fetchErrorDetails(
+      userId: String,
+      sessionId: String,
+      errorId: String): FetchErrorDetailsResponse = {
+    val promise = Promise[FetchErrorDetailsResponse]
+    val handler =
+      new SparkConnectFetchErrorDetailsHandler(new FetchErrorDetailsResponseObserver(promise))
+    val context = proto.UserContext
+      .newBuilder()
+      .setUserId(userId)
+      .build()
+    val request = proto.FetchErrorDetailsRequest
+      .newBuilder()
+      .setUserContext(context)
+      .setSessionId(sessionId)
+      .setErrorId(errorId)
+      .build()
+    handler.handle(request)
+    ThreadUtils.awaitResult(promise.future, 5.seconds)
+  }
+
+  test("error chain is properly constructed") {
+    val testError =
+      new Exception("test1", new Exception("test2"))
+    val errorId = UUID.randomUUID().toString()
+
+    SparkConnectService
+      .getOrCreateIsolatedSession(userId, sessionId)
+      .errorIdToError
+      .put(errorId, testError)
+
+    val response = fetchErrorDetails(userId, sessionId, errorId)
+    assert(response.hasRootErrorIdx)
+    assert(response.getRootErrorIdx == 0)
+
+    assert(response.getErrorsCount == 2)
+    assert(response.getErrors(0).getMessage == "test1")
+    assert(response.getErrors(0).getErrorTypeHierarchy(0) == classOf[Exception].getName)

Review Comment:
   What is the hierarchy for this? I'd expect `Throwable` and `Exception`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330634306


##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {

Review Comment:
   Shouldn't you test feature flags?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330581451


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +214,21 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =

Review Comment:
   TBH I think we should get rid of this feature flag in SPARK-4.1.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329222692


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
 
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
+  private def exceptionInfoToThrowable(exceptionInfo: ExceptionInfo): Option[Throwable] = {
+    val classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala
 
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
+    if (classHierarchy.isEmpty) {
+      return None
+    }
 
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
+    val constructor =
+      classHierarchy
+        .flatMap(errorFactory.get)
+        .headOption
+        .getOrElse((message: String, cause: Option[Throwable]) =>
+          new SparkException(
+            s"Exception in server ${classHierarchy.head}: ${message}",
+            cause.orNull))
+
+    val causeOpt = if (exceptionInfo.hasCause) {
+      exceptionInfoToThrowable(exceptionInfo.getCause)
+    } else {
+      None
+    }

Review Comment:
   What are the failure conditions here? What happens if you cant find the exception you're looking for? Do you have tests for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330172697


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   It is a field in the fetch request for non-jvm client (like python client) to include stack trace in the error message directly. Otherwise, the client needs to convert the structured stack trace data to string



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330247802


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          isTruncated = false,
+          serverStackTraceEnabled =
+            sessionHolder.session.conf.get(Connect.CONNECT_SERVER_STACKTRACE_ENABLED),
+          pySparkJVMStackTraceEnabled =
+            sessionHolder.session.conf.get(SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),

Review Comment:
   If we merge this into one flag, the scala client may be affected by PYSPARK_JVM_STACKTRACE_ENABLED as well. But I think it is probably ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310876196


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(

Review Comment:
   I renamed the stub to grpcStub



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312526514


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {

Review Comment:
   Other remark. Why is this thing returning a list of Error? Why not properly rebuild the structure, and make `Error` have a `cause` and maybe `suppressed` errors. Now you need to do this later on and this makes things a bit more complicated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312522143


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.

Review Comment:
   Please explain how the exception conversion takes place, and this will fallback when it cannot convert an exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310939456


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)

Review Comment:
   The idea is to make the ErrorInfo self-contained for issuing the follow up FetchErrorDetails request. Otherwise, it may make the implementation on the client side a bit more complicate since each FetchErrorDetails call needs to figure out the corresponding session id and user id



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -201,6 +201,20 @@ class SparkConnectService(debug: Boolean) extends AsyncService with BindableServ
         sessionId = request.getSessionId)
   }
 
+  override def getErrorInfo(
+      request: proto.GetErrorInfoRequest,
+      responseObserver: StreamObserver[proto.GetErrorInfoResponse]): Unit = {
+    try {
+      new SparkConnectGetErrorInfoHandler(responseObserver).handle(request)
+    } catch {
+      ErrorUtils.handleError(
+        "getErrorInfo",
+        observer = responseObserver,
+        userId = request.getUserContext.getUserId,
+        sessionId = request.getSessionId)

Review Comment:
   https://github.com/apache/spark/pull/42377#discussion_r1310939456



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309579141


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +183,38 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(status.getMessage, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      errorId: String,
+      classes: Array[String],
+      message: String,
+      stackTraceOpt: Option[Array[StackTraceElement]])
+
+  private def errorsToThrowable(errors: Array[Error]): Option[Throwable] = {
+    if (errors.isEmpty) {
+      return None
+    }
+    if (errors.head.classes.isEmpty) {
+      return None
     }
+    val constructor =
+      errors.head.classes
+        .find(errorFactory.contains)

Review Comment:
   this would save a map lookup: `errors.head.classes.iterator.flatmap(errorFactory.get).headOption.getOrElse(...)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310956914


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +44,40 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test(s"throw SparkException with large cause exception") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {

Review Comment:
   Do you mean [this connect](https://github.com/apache/spark/blob/171251518b469824589c498c5f202cc55dacb128/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala)? It is in `connect/server`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329202438


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -265,6 +271,12 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
 
 object SessionHolder {
 
+  // The maxmimum number of errors in the cache.
+  private val ERROR_CACHE_SIZE = 20
+
+  // The maximum time for an error to stay in the cache.
+  private val ERROR_CACHE_TIMEOUT_SEC = 60
+

Review Comment:
   is this 20 distinct exceptions or 20 stacks?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329352625


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(

Review Comment:
   Will add. The reason I didn't add doc here is because I was trying to be consistent with the document style under this package (other handlers don't have docs as well)... 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329225003


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
 
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
+  private def exceptionInfoToThrowable(exceptionInfo: ExceptionInfo): Option[Throwable] = {

Review Comment:
   this deservers some documentation as well please



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove the change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330443354


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I think we can change to 
   ```
   message Response {
     optional int32 root_error_idx = 1;
   
     repeated Error errors = 2;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330569555


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +49,15 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  // The cache that maps an error id to a throwable. The throwable in cache is independent to
+  // each other.
+  val errorIdToError = CacheBuilder

Review Comment:
   Why is this public? I'd expect style checks to complain about a missing type annotation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330612639


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   How would this perform better than materializing the value after you have checked the config?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330495612


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -91,35 +91,40 @@ private[connect] object ErrorUtils extends Logging {
   private[connect] def throwableToFetchErrorDetailsResponse(
       st: Throwable,
       serverStackTraceEnabled: Boolean = false): FetchErrorDetailsResponse = {
-    val errorChain = traverseCauses(st).take(MAX_ERROR_CHAIN_LENGTH).map { case error =>
-      val builder = FetchErrorDetailsResponse.Error
-        .newBuilder()
-        .setMessage(error.getMessage)
-        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
-
-      if (serverStackTraceEnabled) {
-        builder.addAllStackTrace(
-          error.getStackTrace
-            .map { stackTraceElement =>
-              FetchErrorDetailsResponse.StackTraceElement
-                .newBuilder()
-                .setDeclaringClass(stackTraceElement.getClassName)
-                .setMethodName(stackTraceElement.getMethodName)
-                .setFileName(stackTraceElement.getFileName)
-                .setLineNumber(stackTraceElement.getLineNumber)
-                .build()
-            }
-            .toIterable
-            .asJava)
-      }
+    val errors =
+      traverseCauses(st).take(MAX_ERROR_CHAIN_LENGTH).zipWithIndex.map { case (error, idx) =>
+        val builder = FetchErrorDetailsResponse.Error
+          .newBuilder()
+          .setMessage(error.getMessage)
+          .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+        if (idx + 1 < MAX_ERROR_CHAIN_LENGTH) {
+          builder.setCauseIdx(idx + 1)

Review Comment:
   You should check if it actually has a cause instead of assuming it has one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330152653


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   lead by example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330211093


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   Let's defer this discussion to the client-side change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323060159


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 (TID 55303014) (10.139.68.71 executor 781): org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:244)
   	at org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
   	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 (TID 55303014) (10.139.68.71 executor 781): org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   	at org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:244)
   	at org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
   	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
   org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   java.lang.RuntimeException: Number 2 encountered!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323060159


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   1. org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage ...
   2. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 ...
   3. org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   4. java.lang.RuntimeException: Number 2 encountered!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301851158


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   `org.apache.spark.sql.connect.config.Connect` cannot be configured by `spark.conf.set(...)` if I understand correctly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301852092


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +44,40 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test(s"throw SparkException with large cause exception") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {

Review Comment:
   No if I understand correctly since Connect config is in the server module and client module does not depend on the server module



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323024042


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +218,34 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      classHierarchy: Array[String],
+      message: String,
+      stackTraceOpt: Option[Array[StackTraceElement]],
+      causeOpt: Option[Error])
+
+  private def errorToThrowable(error: Error): Option[Throwable] = {

Review Comment:
   The error framework is for improving the quality of error messages. Currently, we directly construct exception by message and cause and don't retain the relevant fields like errorClass, query context and so on. The current implementation is extensible to the error framework and I will follow up on this to cover the relevant use cases



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322930529


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +45,53 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test("cause exceptions should work without error enrichment") {
+    withSQLConf("spark.sql.connect.enrichError.enabled" -> "false") {
+      val ex = intercept[SparkUpgradeException] {
+        spark
+          .sql("""
+            |select from_json(
+            |  '{"d": "02-29"}',
+            |  'd date',
+            |  map('dateFormat', 'MM-dd'))
+            |""".stripMargin)
+          .collect()
+      }
+      assert(ex.getCause.isInstanceOf[DateTimeException])
+    }
+  }
+
+  test("throw SparkException with large cause exception") {
+    val session = spark
+    import session.implicits._
+
+    val throwException =
+      udf((_: String) => throw new SparkException("test" * 10000))
+
+    val ex = intercept[SparkException] {
+      Seq("1").toDS.withColumn("udf_val", throwException($"value")).collect()
+    }
+
+    assert(ex.getCause.isInstanceOf[SparkException])
+    assert(ex.getCause.getMessage.contains("test" * 10000))

Review Comment:
   what would normally happen?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plans, they are less likely to be very deep. Also, limit on the depth of exceptions are probably more acceptable as long as there is no correctness issue), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions IMO... I can change it back if needed...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plans, they are less likely to be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions IMO... I can change it back if needed...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330578327


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +49,15 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  // The cache that maps an error id to a throwable. The throwable in cache is independent to
+  // each other.
+  val errorIdToError = CacheBuilder

Review Comment:
   Good catch, it shouldn't... 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330155642


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -291,15 +307,16 @@ object SparkConnectService extends Logging {
   }
 
   // Simple builder for creating the cache of Sessions.
-  private def cacheBuilder(cacheSize: Int, timeoutSeconds: Int): CacheBuilder[Object, Object] = {
+  private[service] def cacheBuilder(

Review Comment:
   Why do we have the crossing concerns here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329226769


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))
+          .map(ProtoAny.pack(_))
+          .asJava)
+      .setMessage(SparkConnectService.extractErrorMessage(st, MAX_MESSAGE_SIZE))

Review Comment:
   This is a change to the previous size of the message. We used to have 2048 bytes, now we have 512.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329225789


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -291,15 +307,16 @@ object SparkConnectService extends Logging {
   }
 
   // Simple builder for creating the cache of Sessions.
-  private def cacheBuilder(cacheSize: Int, timeoutSeconds: Int): CacheBuilder[Object, Object] = {
+  private[service] def cacheBuilder(

Review Comment:
   Is the vis change needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the RPC failure).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329204755


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5

Review Comment:
   Doc?



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -31,16 +34,25 @@ import org.apache.commons.lang3.exception.ExceptionUtils
 import org.json4s.JsonDSL._
 import org.json4s.jackson.JsonMethods
 
-import org.apache.spark.{SparkEnv, SparkException, SparkThrowable}
+import org.apache.spark.{SparkException, SparkThrowable}
 import org.apache.spark.api.python.PythonException
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
 import org.apache.spark.internal.Logging
 import org.apache.spark.sql.connect.config.Connect
 import org.apache.spark.sql.connect.service.ExecuteEventsManager
 import org.apache.spark.sql.connect.service.SparkConnectService
-import org.apache.spark.sql.internal.SQLConf
 
 private[connect] object ErrorUtils extends Logging {

Review Comment:
   doc?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(

Review Comment:
   I will remove it then...



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC fails (maybe some bugs on the server side, the exception is no longer cached), fallback to an abbreviated FetchErrorDetailsResponse is more consistent with the existing control flow. Otherwise, we may have to throw a different Exception or fallback to an exception without causes (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300383


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   This is for Python Client. Unlike Scala client, Python client (or non-jvm client) does not support constructing jvm stacktrace from structured data, so we should render the message with stacktrace directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300383


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   This is for Python Client. Unlike Scala client, Python client (or non-jvm client) does not support constructing jvm stacktrace from structured data, so we render the message with stacktrace directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330169939


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +46,13 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  // The cache that maps an error id to a throwable. The throwable in cache is independent to
+  // each other.
+  val errorIdToError =
+    SparkConnectService
+      .cacheBuilder(ERROR_CACHE_SIZE, ERROR_CACHE_TIMEOUT_SEC)

Review Comment:
   LRU. But errors will be invalidated after the first fetch request



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309562656


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   @heyihong if you want to set the conf without getting yelled at by SQLConf, use `onCreate(SQLConf.register)` when you are building the conf.
   
   @grundprinzip that would make it a dynamic flag right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309581910


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,38 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {
+      val session = spark
+      import session.implicits._
+
+      val checkForTwo = udf((value: Int) => {
+        if (value == 2) {
+          throw new RuntimeException("Number 2 encountered!")
+        }
+        value
+      })
+
+      val query = spark.readStream
+        .format("rate")
+        .option("rowsPerSecond", "1")
+        .load()
+        .select(checkForTwo($"value").as("checkedValue"))
+        .writeStream
+        .outputMode("append")
+        .format("console")
+        .start()
+
+      val exception = intercept[SparkException] {

Review Comment:
   Shouldn't this be a StreamingQueryException?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300383


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   This is for Python Client. Unlike Scala client, Python client (or non-jvm client) does not support constructing jvm stacktrace from structured data very well, so we should render the message with stacktrace directly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1685594500

   Would be great if we have the user-facing exception (and stacktrace) example at the PR description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1300891626


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   Would have to be `4.0.0` - I think it's too late for 3.5.0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300383


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   This is for Python Client



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329285329


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -265,6 +271,12 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
 
 object SessionHolder {
 
+  // The maxmimum number of errors in the cache.
+  private val ERROR_CACHE_SIZE = 20
+
+  // The maximum time for an error to stay in the cache.
+  private val ERROR_CACHE_TIMEOUT_SEC = 60
+

Review Comment:
   20 distinct exceptions. Will update the comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(

Review Comment:
   I will remove it then...
   
   The issue here is that unlike Python or other non-jvm client, they can directly fallback to use message and classes in ErrorInfo . For scala client, doing such fallback means losing cause exceptions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329325417


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   It is possible if a client with newer version connects to a server with older version



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329210865


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   Why do you need to send an abbreviated FetchErrorDetailsResponse here? We did say that the user has to explicitly, fetch the exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329219527


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   This comes with a couple of disadvantages: 
   
   Recursive nesting is bad for protobuf and we have explicit nesting depths. If we can we should consider avoiding that.
   
   Why is this not a problem? 
   
   Nested exceptions do not form a a tree but rather just a "stack". This means the easiest way to represent a list of exceptions is using a list. The first element in this list would be the top most exception, whereas the last one is the one that actually triggered the first one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329214384


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,

Review Comment:
   This is confusing again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1724508214

   > With all the comments etc the PR becomes quite large and my suggestion would be to split it into two. One for the server-side changes and one for the client-side changes.
   
   The server side change will affect the python client behavior since I remove the JVM stacktrace from ErrorInfo. Below is my proposed split:
   1. Server-side change + Python Client change
   2. Scala Client change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329208820


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.

Review Comment:
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the additional RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330443354


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * Handles [[proto.FetchErrorDetailsRequest]]s for the [[SparkConnectService]].
+ *
+ * @param responseObserver
+ */
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        sessionHolder.errorIdToError.invalidate(v.getErrorId)
+
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          serverStackTraceEnabled = sessionHolder.session.conf.get(
+            Connect.CONNECT_SERVER_STACKTRACE_ENABLED) || sessionHolder.session.conf.get(
+            SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),
+          stackTraceInMessage = v.getStacktraceInMessage)
+      }
+      .getOrElse(FetchErrorDetailsResponse.newBuilder().build())

Review Comment:
   I think we can change to 
   ```
   message Response {
     optional int32 root_error_idx = 1;
   
     repeated Error errors = 2;
   }
   ```
   
   We can use the root_error_idx to indicate whether the error exists
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330150762


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          isTruncated = false,
+          serverStackTraceEnabled =
+            sessionHolder.session.conf.get(Connect.CONNECT_SERVER_STACKTRACE_ENABLED),
+          pySparkJVMStackTraceEnabled =
+            sessionHolder.session.conf.get(SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),

Review Comment:
   Why can't you merge this into one flag. You could even have two outside, but why does this need to be two here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1300893521


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +44,40 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test(s"throw SparkException with large cause exception") {
+    withSQLConf("spark.sql.spark.enrichError.enabled" -> "true") {

Review Comment:
   Can we use `Connect.SPARK_ENRICH_ERROR_ENABLED.key` to get the key



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301851158


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   `org.apache.spark.sql.connect.config.Connect` cannot be configured by `spark.conf.set(...)` if I understand correctly?
   
   Maybe we can name this to `spark.sql.connect.enrichError.enabled`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1311496078


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -813,5 +869,8 @@ service SparkConnectService {
   // Non reattachable executions are released automatically and immediately after the ExecutePlan
   // RPC and ReleaseExecute may not be used.
   rpc ReleaseExecute(ReleaseExecuteRequest) returns (ReleaseExecuteResponse) {}
+
+  // GetErrorInfo retrieves cached exceptions based on provided error ids.
+  rpc GetErrorInfo(GetErrorInfoRequest) returns (GetErrorInfoResponse) {}

Review Comment:
   Maybe we can use FetchErrorDetails instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310947001


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetErrorInfoHandler.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import scala.jdk.CollectionConverters._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.GetErrorInfoResponse
+import org.apache.spark.connect.proto.GetErrorInfoResponse.{ExceptionInfo, StackTraceElementInfo}
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectGetErrorInfoHandler(
+    responseObserver: StreamObserver[proto.GetErrorInfoResponse]) {
+
+  def handle(v: proto.GetErrorInfoRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val exceptionInfos = v.getErrorIdsList.asScala.flatMap { errorId =>
+      Option(sessionHolder.errorIdToError.getIfPresent(errorId)) match {
+        case Some(error) =>
+          // Invalidate the cache immediately to free up space for other exceptions to be stored
+          // since this exception will no longer be accessed.
+          sessionHolder.errorIdToError.invalidate(errorId)

Review Comment:
   This can be a very rare case. Are we concerning about the UUID in general or for this case specifically? We are using UUID in a lot of other places as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1313864858


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =
+    buildStaticConf("spark.connect.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)
+
+  val CONNECT_SERVER_STACKTRACE_ENABLED =

Review Comment:
   Same for the performance reason (it is disabled by default)



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =

Review Comment:
   Configureable per session may be better since turning off the flag (it is enabled by default) can make the error handling faster (i.e. without an additional rpc)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330446640


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   They don't alway form a stack if you use suppressed exceptions. In that case an exception would point to a cause, and a number of suppressed exceptions. Sometimes the suppressed exception is the one you want, and not the cause.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322931419


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   How is this thing wrapped? What is the structure?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301913064


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   > Would have to be `4.0.0` - I think it's too late for 3.5.0.
   
   Can `3.5.0` have this pr with the feature disabled by default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301851158


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   `org.apache.spark.sql.connect.config.Connect` cannot be configured by `spark.conf.set(...)` if I understand correctly?
   
   Maybe we can name this to `spark.sql.connect.jvmStacktrace.enabled`



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   `org.apache.spark.sql.connect.config.Connect` cannot be configured by `spark.conf.set(...)` if I understand correctly?
   
   Maybe we can name this to `spark.sql.spark.enrichError.enabled`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1308595431


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;

Review Comment:
   Does it have to be a repeated here?



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {
+    // Unique identifier for this specific exception instance.
+    string error_id = 1;
+
+    // Fully qualified names of the exception class and its parental classes.
+    repeated string error_type_hierarchy = 2;
+
+    // Detailed message of the exception.
+    string message = 3;
+
+    // StackTrace of the exception.
+    repeated StackTraceElementInfo stack_trace = 4;
+  }
+
+  // List of matched exceptions.
+  repeated ExceptionInfo exception_infos = 1;

Review Comment:
   Does this need a comment on the order of the trace elements?



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {

Review Comment:
   In Java the element is called `StackTraceElement`, my proposal would be to name it like this as well. In addition, I'm wondering how much benefit we get from this custom serialization because the `toString` implementation already is able to deal with the edge cases of absent line numbers etc
   
   https://docs.oracle.com/javase/8/docs/api/java/lang/StackTraceElement.html#toString--



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CustomSparkConnectBlockingStub.scala:
##########
@@ -27,11 +27,14 @@ private[client] class CustomSparkConnectBlockingStub(
     retryPolicy: GrpcRetryHandler.RetryPolicy) {
 
   private val stub = SparkConnectServiceGrpc.newBlockingStub(channel)
+
   private val retryHandler = new GrpcRetryHandler(retryPolicy)
 
+  private val grpcExceptionConverter = new GrpcExceptionConverter(stub)

Review Comment:
   doc? Why does this need the stub now?



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {

Review Comment:
   doc



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -40,7 +42,15 @@ import org.apache.spark.sql.connect.service.SparkConnectService
 import org.apache.spark.sql.internal.SQLConf
 
 private[connect] object ErrorUtils extends Logging {
-  private def allClasses(cl: Class[_]): Seq[Class[_]] = {
+  private[connect] def allThrowables(t: Throwable): List[Throwable] = {

Review Comment:
   ```suggestion
     private[connect] def traverseCauses(t: Throwable): List[Throwable] = {
   ```
   
   maybe better?



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {

Review Comment:
   Can you explain a little bit more about the uniqueness properties of the error ID? Is there guaranteed to be exactly one?



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -813,5 +869,8 @@ service SparkConnectService {
   // Non reattachable executions are released automatically and immediately after the ExecutePlan
   // RPC and ReleaseExecute may not be used.
   rpc ReleaseExecute(ReleaseExecuteRequest) returns (ReleaseExecuteResponse) {}
+
+  // GetErrorInfo retrieves cached exceptions based on provided error ids.
+  rpc GetErrorInfo(GetErrorInfoRequest) returns (GetErrorInfoResponse) {}

Review Comment:
   ```suggestion
     rpc FetchExecutionErrorDetails(FetchExecutionErrorDetailsRequest) returns (FetchExecutionErrorDetailsResponse) {}
   ```



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,18 +67,62 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private val NUM_ERRORS_LIMIT = 5
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      stackTraceEnabled: Boolean,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
+      .putMetadata("classes", serializeClasses(st))
+
+    // We can get full exception messages and optionally stacktrace by
+    // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+    // limit to reduce the probability of hitting the 8KB header limit.
+    val metadataValueLimit = if (enrichErrorEnabled) 512 else Int.MaxValue
+    val maxMessageSize = Math.min(metadataValueLimit, 2048)
+
+    if (enrichErrorEnabled) {
+      val errorIdAndErrors =
+        allThrowables(st).map((UUID.randomUUID().toString, _)).take(NUM_ERRORS_LIMIT)

Review Comment:
   it might be irrelevant, but `allThrowables(st).take(NUM_ERRORS_LIMIT).map()` avoids generating UUIDs that you don't need.



##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   Should this actually be a SQL conf? Shouldn't this be a static server conf on the server and the client can still turn off if desired.



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetErrorInfoHandler.scala:
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import scala.jdk.CollectionConverters._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.GetErrorInfoResponse
+import org.apache.spark.connect.proto.GetErrorInfoResponse.{ExceptionInfo, StackTraceElementInfo}
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectGetErrorInfoHandler(
+    responseObserver: StreamObserver[proto.GetErrorInfoResponse]) {
+
+  def handle(v: proto.GetErrorInfoRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val exceptionInfos = v.getErrorIdsList.asScala.flatMap { errorId =>
+      Option(sessionHolder.errorIdToError.getIfPresent(errorId)) match {
+        case Some(error) =>
+          // Invalidate the cache immediately to free up space for other exceptions to be stored
+          // since this exception will no longer be accessed.
+          sessionHolder.errorIdToError.invalidate(errorId)

Review Comment:
   It might be an unlikely case, but the writer should check if an entry already exists for the UUID and in the worst case generate a new one.



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {
+    // Unique identifier for this specific exception instance.
+    string error_id = 1;

Review Comment:
   if error id I unique, shouldn't the exception infos be a map instead?



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -24,15 +24,20 @@ import scala.reflect.ClassTag
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{GetErrorInfoRequest, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
+private[client] class GrpcExceptionConverter(stub: SparkConnectServiceBlockingStub) {

Review Comment:
   None of this code here has any appropriate documentation. Since the code was originally written by you, I suggest that you provide proper documentation here as well.



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(

Review Comment:
   It's not obvious directly that this is doing an RPC now. 



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)

Review Comment:
   Session ID is a client specified value, similar to the User ID, we should not have the need to return them to the client.



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,18 +67,62 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private val NUM_ERRORS_LIMIT = 5
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      stackTraceEnabled: Boolean,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
+      .putMetadata("classes", serializeClasses(st))
+
+    // We can get full exception messages and optionally stacktrace by
+    // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+    // limit to reduce the probability of hitting the 8KB header limit.
+    val metadataValueLimit = if (enrichErrorEnabled) 512 else Int.MaxValue
+    val maxMessageSize = Math.min(metadataValueLimit, 2048)
+
+    if (enrichErrorEnabled) {
+      val errorIdAndErrors =
+        allThrowables(st).map((UUID.randomUUID().toString, _)).take(NUM_ERRORS_LIMIT)
+
+      errorInfo
+        .putMetadata("userId", userId)
+        .putMetadata("sessionId", sessionId)
+        .putMetadata("errorId", errorIdAndErrors.head._1)
+        .putMetadata(
+          "causes",
+          JsonMethods.compact(JsonMethods.render(errorIdAndErrors.tail.map {
+            case (errorId, error) =>
+              Map(
+                "errorId" -> errorId,
+                "message" -> SparkConnectService.extractErrorMessage(error, maxMessageSize),
+                "classes" -> serializeClasses(error))
+          })))
+
+      val sessionHolder =
+        SparkConnectService
+          .getOrCreateIsolatedSession(userId, sessionId)
+
+      errorIdAndErrors.foreach { case (errorId, error) =>
+        sessionHolder.errorIdToError.put(errorId, error)
+      }
+    }
 
     lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
     val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {

Review Comment:
   adding the new RPC should never embedd the stack trace into the response, but always fetch the stack trace using the additional RPC.



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -201,6 +201,20 @@ class SparkConnectService(debug: Boolean) extends AsyncService with BindableServ
         sessionId = request.getSessionId)
   }
 
+  override def getErrorInfo(
+      request: proto.GetErrorInfoRequest,
+      responseObserver: StreamObserver[proto.GetErrorInfoResponse]): Unit = {
+    try {
+      new SparkConnectGetErrorInfoHandler(responseObserver).handle(request)
+    } catch {
+      ErrorUtils.handleError(
+        "getErrorInfo",
+        observer = responseObserver,
+        userId = request.getUserContext.getUserId,
+        sessionId = request.getSessionId)

Review Comment:
   There should be no need to return them from the server



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,18 +67,62 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private val NUM_ERRORS_LIMIT = 5
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      stackTraceEnabled: Boolean,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
+      .putMetadata("classes", serializeClasses(st))
+
+    // We can get full exception messages and optionally stacktrace by
+    // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+    // limit to reduce the probability of hitting the 8KB header limit.
+    val metadataValueLimit = if (enrichErrorEnabled) 512 else Int.MaxValue
+    val maxMessageSize = Math.min(metadataValueLimit, 2048)
+
+    if (enrichErrorEnabled) {
+      val errorIdAndErrors =
+        allThrowables(st).map((UUID.randomUUID().toString, _)).take(NUM_ERRORS_LIMIT)
+
+      errorInfo
+        .putMetadata("userId", userId)
+        .putMetadata("sessionId", sessionId)
+        .putMetadata("errorId", errorIdAndErrors.head._1)

Review Comment:
   I think this is the part where the whole documentation needs to be much more explicit. What does the errorID here actually mean compared to the other error IDs.



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -201,6 +201,20 @@ class SparkConnectService(debug: Boolean) extends AsyncService with BindableServ
         sessionId = request.getSessionId)
   }
 
+  override def getErrorInfo(
+      request: proto.GetErrorInfoRequest,
+      responseObserver: StreamObserver[proto.GetErrorInfoResponse]): Unit = {
+    try {
+      new SparkConnectGetErrorInfoHandler(responseObserver).handle(request)
+    } catch {
+      ErrorUtils.handleError(
+        "getErrorInfo",
+        observer = responseObserver,
+        userId = request.getUserContext.getUserId,
+        sessionId = request.getSessionId)

Review Comment:
   user and session ID are generated by the client
   



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +45,9 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  val errorIdToError =

Review Comment:
   doc on the magic number params. Make them configurable?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309568509


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(
+        GetErrorInfoRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .addAllErrorIds(errors.map(_.errorId).toIterable.asJava)
+          .setUserContext(UserContext.newBuilder().setUserId(userId).build())
+          .build())
+
+      val exceptionInfos = errorInfoResponse.getExceptionInfosList.asScala
+
+      errors.map { error =>
+        exceptionInfos.find(_.getErrorId == error.errorId) match {
+          case Some(exceptionInfo) =>
+            val stackTraceOpt =
+              if (exceptionInfo.getStackTraceList.isEmpty) None
+              else {
+                Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+                  new StackTraceElement(
+                    stackTraceElement.getDeclaringClass,
+                    stackTraceElement.getMethodName,
+                    stackTraceElement.getFileName,
+                    stackTraceElement.getLineNumber)
+                })
+              }
+            Error(
+              errorId = error.errorId,
+              classes = error.classes,
+              message = exceptionInfo.getMessage,
+              stackTraceOpt = stackTraceOpt)
+          case None =>
+            error
+        }
+      }
+    } catch {
+      case _: StatusRuntimeException => errors

Review Comment:
   Do we want to retry here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1313613252


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {
+    exceptionInfos.get(errorId) match {
+      case Some(exceptionInfo) =>
+        val stackTraceOpt =
+          if (exceptionInfo.getStackTraceList.isEmpty) None
+          else {
+            Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+              new StackTraceElement(
+                stackTraceElement.getDeclaringClass,
+                stackTraceElement.getMethodName,
+                stackTraceElement.getFileName,
+                stackTraceElement.getLineNumber)
+            })
+          }
+        val causeErrors = if (exceptionInfo.hasCauseErrorId) {
+          exceptionInfosToErrors(exceptionInfo.getCauseErrorId, exceptionInfos)
+        } else {
+          List.empty
+        }
+        Error(
+          classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala.toArray,
+          message = exceptionInfo.getMessage,
+          stackTraceOpt = stackTraceOpt
+        ) +: causeErrors
+      case None =>
+        List.empty

Review Comment:
   https://github.com/apache/spark/pull/42377#discussion_r1313613119



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {
+    exceptionInfos.get(errorId) match {
+      case Some(exceptionInfo) =>
+        val stackTraceOpt =
+          if (exceptionInfo.getStackTraceList.isEmpty) None
+          else {
+            Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+              new StackTraceElement(
+                stackTraceElement.getDeclaringClass,
+                stackTraceElement.getMethodName,
+                stackTraceElement.getFileName,
+                stackTraceElement.getLineNumber)
+            })
+          }
+        val causeErrors = if (exceptionInfo.hasCauseErrorId) {
+          exceptionInfosToErrors(exceptionInfo.getCauseErrorId, exceptionInfos)
+        } else {
+          List.empty

Review Comment:
   https://github.com/apache/spark/pull/42377#discussion_r1313613119



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1313864811


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =

Review Comment:
   Configureable per session may be better since turning off the flag can make the error handling faster (i.e. without an additional rpc)



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =
+    buildStaticConf("spark.connect.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)
+
+  val CONNECT_SERVER_STACKTRACE_ENABLED =

Review Comment:
   Same for the performance reason



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330160045


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   The stack trace is not complicated algorithmic logic. Rendering this on the client should not be an issue. Having duplication of stacktrace_in_message does not make sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330216449


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))
+          .map(ProtoAny.pack(_))
+          .asJava)
+      .setMessage(SparkConnectService.extractErrorMessage(st, MAX_MESSAGE_SIZE))

Review Comment:
   I undo the change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330178093


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   For example, if we want to include the stack trace of cause exceptions in the error messages (currently only the stacktrace of the root exception), there are more works for all non-jvm clients 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330174497


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   And why is this bad?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330495974


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   So you can suppress multiple? Do you have an example handy? Should we make this part of the structure to list the surpessed exceptions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plans, they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions IMO... I can change it back if needed...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329224205


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
 
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
+  private def exceptionInfoToThrowable(exceptionInfo: ExceptionInfo): Option[Throwable] = {
+    val classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala
 
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
+    if (classHierarchy.isEmpty) {
+      return None
+    }
 
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
+    val constructor =
+      classHierarchy
+        .flatMap(errorFactory.get)
+        .headOption
+        .getOrElse((message: String, cause: Option[Throwable]) =>
+          new SparkException(
+            s"Exception in server ${classHierarchy.head}: ${message}",

Review Comment:
   Exception received from Spark Connect on server: ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(

Review Comment:
   I will remove it then...
   
   The issue here is that unlike Python or other non-jvm client, they can directly fallback to use message and classes in ErrorInfo if the RPC fails. For scala client, doing such fallback means losing cause exceptions that may affect control flows



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1726931731

   Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330626485


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   I think it is trying to use some Short-circuit evaluation if I understand correctly... But I don't think it is necessary...



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   It is trying to use some Short-circuit evaluation if I understand correctly... But I don't think it is necessary...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323024042


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +218,34 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      classHierarchy: Array[String],
+      message: String,
+      stackTraceOpt: Option[Array[StackTraceElement]],
+      causeOpt: Option[Error])
+
+  private def errorToThrowable(error: Error): Option[Throwable] = {

Review Comment:
   The error framework is for improving the quality of error messages. Currently, we directly construct exception by message and cause and don't retain the relevant fields like errorClass, query context and so on. The current implementation should be extensible to the error framework and I will follow up on this to cover the relevant use cases



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322665674


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -24,46 +24,174 @@ import scala.reflect.ClassTag
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfoToError(exceptionInfo: ExceptionInfo): Option[Error] = {
+    val stackTraceOpt =
+      if (exceptionInfo.getStackTraceList.isEmpty) None
+      else {
+        Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+          new StackTraceElement(
+            stackTraceElement.getDeclaringClass,
+            stackTraceElement.getMethodName,
+            stackTraceElement.getFileName,
+            stackTraceElement.getLineNumber)
+        })
+      }
+    Some(
+      Error(
+        classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala.toArray,
+        message = exceptionInfo.getMessage,
+        stackTraceOpt = stackTraceOpt,
+        causeOpt = if (exceptionInfo.hasCause) {
+          exceptionInfoToError(exceptionInfo.getCause)
+        } else {
+          None
+        }))
+  }
+
+  /**
+   * enrichError enrich errors with full exception message and optionally stacktrace by issuing an
+   * additional RPC call to fetch error details. The RPC call is best-effort at-most-once.
+   */
+  private def enrichError(
+      error: Error,
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Error = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return error
+    }
+
+    try {
+      val errorDetailsResponse = grpcStub.fetchErrorDetails(
+        FetchErrorDetailsRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .setErrorId(errorId)
+          .setUserContext(UserContext.newBuilder().setUserId(userContext.getUserId).build())
+          .build())
+
+      if (!errorDetailsResponse.hasExceptionInfo) {
+        return error
+      }
+
+      exceptionInfoToError(errorDetailsResponse.getExceptionInfo).get
+    } catch {
+      case _: StatusRuntimeException => error
+    }
+  }
+
+  private def errorInfoToThrowable(
+      info: ErrorInfo,
+      truncatedMessage: String,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val classes = deserializeClasses(info.getMetadataOrDefault("classes", "[]"))
+
+    implicit val formats = DefaultFormats
+    var causeOpt: Option[Error] = None
+    JsonMethods
+      .parse(info.getMetadataOrDefault("causes", "[]"))
+      .extract[Array[Map[String, String]]]
+      .reverse
+      .foreach { m =>
+        causeOpt = Some(
+          Error(
+            classHierarchy = deserializeClasses(m.get("classes").get),
+            message = m.get("message").get,
+            stackTraceOpt = None,
+            causeOpt = causeOpt))
+      }
+    val error = Error(classes, truncatedMessage, stackTraceOpt = None, causeOpt = causeOpt)
+    errorToThrowable(enrichError(error, info, sessionId, userContext))

Review Comment:
   Why not first try to enrich and then do the fallback. Now we are duplicate work. And, more importantly, it is much harder to follow, a casual reader might get confused.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322656617


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -24,46 +24,174 @@ import scala.reflect.ClassTag
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfoToError(exceptionInfo: ExceptionInfo): Option[Error] = {
+    val stackTraceOpt =
+      if (exceptionInfo.getStackTraceList.isEmpty) None
+      else {
+        Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+          new StackTraceElement(
+            stackTraceElement.getDeclaringClass,
+            stackTraceElement.getMethodName,
+            stackTraceElement.getFileName,
+            stackTraceElement.getLineNumber)
+        })
+      }
+    Some(
+      Error(
+        classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala.toArray,
+        message = exceptionInfo.getMessage,
+        stackTraceOpt = stackTraceOpt,
+        causeOpt = if (exceptionInfo.hasCause) {
+          exceptionInfoToError(exceptionInfo.getCause)
+        } else {
+          None
+        }))
+  }
+
+  /**
+   * enrichError enrich errors with full exception message and optionally stacktrace by issuing an
+   * additional RPC call to fetch error details. The RPC call is best-effort at-most-once.
+   */
+  private def enrichError(
+      error: Error,
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Error = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return error
+    }
+
+    try {
+      val errorDetailsResponse = grpcStub.fetchErrorDetails(
+        FetchErrorDetailsRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .setErrorId(errorId)
+          .setUserContext(UserContext.newBuilder().setUserId(userContext.getUserId).build())
+          .build())
+
+      if (!errorDetailsResponse.hasExceptionInfo) {
+        return error
+      }
+
+      exceptionInfoToError(errorDetailsResponse.getExceptionInfo).get
+    } catch {
+      case _: StatusRuntimeException => error
+    }
+  }
+
+  private def errorInfoToThrowable(
+      info: ErrorInfo,
+      truncatedMessage: String,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val classes = deserializeClasses(info.getMetadataOrDefault("classes", "[]"))
+
+    implicit val formats = DefaultFormats
+    var causeOpt: Option[Error] = None
+    JsonMethods

Review Comment:
   Please use a `loop` or `foldRight` instead side-effecting like this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329814944


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))
+          .map(ProtoAny.pack(_))
+          .asJava)
+      .setMessage(SparkConnectService.extractErrorMessage(st, MAX_MESSAGE_SIZE))

Review Comment:
   I changed the message size back to 2048 in the splited pr.



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))
+          .map(ProtoAny.pack(_))
+          .asJava)
+      .setMessage(SparkConnectService.extractErrorMessage(st, MAX_MESSAGE_SIZE))

Review Comment:
   I changed the message size back to 2048 in the splitted pr.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330178093


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   For example, if we want to include the stack trace of cause exceptions in the error messages (currently only the stacktrace of the root exception), there are more works for all non-jvm clients (and harder to change due to multi-versions)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330283541


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,6 +146,18 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
+    if (enrichErrorEnabled) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)

Review Comment:
   THe caller has the session holder, why not pass it instead of calling this again?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330163481


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +46,13 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  // The cache that maps an error id to a throwable. The throwable in cache is independent to
+  // each other.
+  val errorIdToError =
+    SparkConnectService
+      .cacheBuilder(ERROR_CACHE_SIZE, ERROR_CACHE_TIMEOUT_SEC)

Review Comment:
   is this an LRU or LIFO cache?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323060159


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   1. org.apache.spark.SparkException: org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage ...
   2. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 ...
   3. org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   4. org.apache.spark.SparkException: java.lang.RuntimeException: Number 2 encountered!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329969680


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,

Review Comment:
   This is for supporting the existing configuration. I added the document for this function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330215015


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   https://github.com/apache/spark/pull/42377#discussion_r1330178093



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   Let's move the discussion to https://github.com/apache/spark/pull/42377#discussion_r1330178093



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330211430


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),

Review Comment:
   Let's defer this discussion to the client-side change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plan, they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions... I can change it back...



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plan, they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal though it may be a little bit overfitting for cause exceptions IMO... I can change it back...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal... I can change it back...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330155084


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(

Review Comment:
   You should write the code for the reader not what you see around you. You're not going to be the last person touching or understanding the code. 
   
   What is the purpose of this handler and what is this used for



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330248359


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -291,15 +307,16 @@ object SparkConnectService extends Logging {
   }
 
   // Simple builder for creating the cache of Sessions.
-  private def cacheBuilder(cacheSize: Int, timeoutSeconds: Int): CacheBuilder[Object, Object] = {
+  private[service] def cacheBuilder(

Review Comment:
   I will isolate them then



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330636049


##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {

Review Comment:
   Or, things like `MAX_ERROR_CHAIN_LENGTH`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1300891376


##########
python/pyspark/errors/exceptions/connect.py:
##########
@@ -45,14 +47,34 @@ class SparkConnectException(PySparkException):
     """
 
 
-def convert_exception(info: "ErrorInfo", message: str) -> SparkConnectException:
+def convert_exception(
+    info: "ErrorInfo", message: str, stub: "SparkConnectServiceStub"
+) -> SparkConnectException:
     classes = []
     if "classes" in info.metadata:
         classes = json.loads(info.metadata["classes"])
 
+    stack_trace = ""
+
     if "stackTrace" in info.metadata:
-        stackTrace = info.metadata["stackTrace"]
-        message += f"\n\nJVM stacktrace:\n{stackTrace}"
+        stack_trace = info.metadata["stackTrace"]
+
+    if "userId" in info.metadata and "sessionId" in info.metadata and "errorId" in info.metadata:
+        req = pb2.GetErrorInfoRequest()
+        req.user_context.user_id = info.metadata["userId"]
+        req.session_id = info.metadata["sessionId"]
+        req.error_ids.append(info.metadata["errorId"])
+
+        try:
+            result = GetErrorInfoResult.fromProto(stub.GetErrorInfo(req))
+            if result is not None and result.error_id == info.metadata["errorId"]:
+                message = result.message
+                stack_trace = result.stack_trace
+        except (Exception,):

Review Comment:
   ```suggestion
           except Exception:
   ```
   
   Which exception do we expect to ignore here? Would be great if we can catch narrower scope.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323060159


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/streaming/ClientStreamingQuerySuite.scala:
##########
@@ -175,6 +175,36 @@ class ClientStreamingQuerySuite extends QueryTest with SQLHelper with Logging {
     }
   }
 
+  test("throw exception in streaming") {
+    val session = spark
+    import session.implicits._
+
+    val checkForTwo = udf((value: Int) => {
+      if (value == 2) {
+        throw new RuntimeException("Number 2 encountered!")
+      }
+      value
+    })
+
+    val query = spark.readStream
+      .format("rate")
+      .option("rowsPerSecond", "1")
+      .load()
+      .select(checkForTwo($"value").as("checkedValue"))
+      .writeStream
+      .outputMode("append")
+      .format("console")
+      .start()
+
+    val exception = intercept[SparkException] {
+      query.awaitTermination()
+    }
+
+    assert(
+      exception.getCause.getCause.getCause.getMessage

Review Comment:
   1. org.apache.spark.sql.streaming.StreamingQueryException (SparkException for now): Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage ...
   2. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1384542.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1384542.0 ...
   3. org.apache.spark.SparkException: [FAILED_EXECUTE_UDF] Failed to execute user defined function (`$Lambda$19844/1935914328`: (int) => int).
   4. java.lang.RuntimeException: Number 2 encountered!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322934577


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =
+    buildStaticConf("spark.connect.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)
+
+  val CONNECT_SERVER_STACKTRACE_ENABLED =

Review Comment:
   I think it is fine to enable this by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1311663301


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/CustomSparkConnectBlockingStub.scala:
##########
@@ -27,11 +27,15 @@ private[client] class CustomSparkConnectBlockingStub(
     retryPolicy: GrpcRetryHandler.RetryPolicy) {
 
   private val stub = SparkConnectServiceGrpc.newBlockingStub(channel)
+
   private val retryHandler = new GrpcRetryHandler(retryPolicy)
 
+  // Constructing GrpcExceptionConverter with a GRPC stub for fetching error details from server.
+  private val grpcExceptionConverter = new GrpcExceptionConverter(stub)
+
   def executePlan(request: ExecutePlanRequest): CloseableIterator[ExecutePlanResponse] = {
-    GrpcExceptionConverter.convert {
-      GrpcExceptionConverter.convertIterator[ExecutePlanResponse](
+    grpcExceptionConverter.convert {
+      grpcExceptionConverter.convertIterator[ExecutePlanResponse](

Review Comment:
   you can get the session ID and User ID from here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312527266


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =

Review Comment:
   Is this a static conf? Or configurable per session?



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =
+    buildStaticConf("spark.connect.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)
+
+  val CONNECT_SERVER_STACKTRACE_ENABLED =

Review Comment:
    Same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1313613119


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {

Review Comment:
   Done



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {

Review Comment:
   Good point, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1313613318


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {

Review Comment:
   https://github.com/apache/spark/pull/42377#discussion_r1313613119



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310143393


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(
+        GetErrorInfoRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .addAllErrorIds(errors.map(_.errorId).toIterable.asJava)
+          .setUserContext(UserContext.newBuilder().setUserId(userId).build())
+          .build())
+
+      val exceptionInfos = errorInfoResponse.getExceptionInfosList.asScala
+
+      errors.map { error =>
+        exceptionInfos.find(_.getErrorId == error.errorId) match {
+          case Some(exceptionInfo) =>
+            val stackTraceOpt =
+              if (exceptionInfo.getStackTraceList.isEmpty) None
+              else {
+                Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+                  new StackTraceElement(
+                    stackTraceElement.getDeclaringClass,
+                    stackTraceElement.getMethodName,
+                    stackTraceElement.getFileName,
+                    stackTraceElement.getLineNumber)
+                })
+              }
+            Error(
+              errorId = error.errorId,
+              classes = error.classes,
+              message = exceptionInfo.getMessage,
+              stackTraceOpt = stackTraceOpt)
+          case None =>
+            error
+        }
+      }
+    } catch {
+      case _: StatusRuntimeException => errors

Review Comment:
   The current expectation on this additional rpc call is best effort and at most once so no?



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)
+
+    if (sessionId == null || userId == null) {
+      return errors
+    }
+
+    try {
+      val errorInfoResponse = stub.getErrorInfo(
+        GetErrorInfoRequest
+          .newBuilder()
+          .setSessionId(sessionId)
+          .addAllErrorIds(errors.map(_.errorId).toIterable.asJava)
+          .setUserContext(UserContext.newBuilder().setUserId(userId).build())
+          .build())
+
+      val exceptionInfos = errorInfoResponse.getExceptionInfosList.asScala
+
+      errors.map { error =>
+        exceptionInfos.find(_.getErrorId == error.errorId) match {
+          case Some(exceptionInfo) =>
+            val stackTraceOpt =
+              if (exceptionInfo.getStackTraceList.isEmpty) None
+              else {
+                Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+                  new StackTraceElement(
+                    stackTraceElement.getDeclaringClass,
+                    stackTraceElement.getMethodName,
+                    stackTraceElement.getFileName,
+                    stackTraceElement.getLineNumber)
+                })
+              }
+            Error(
+              errorId = error.errorId,
+              classes = error.classes,
+              message = exceptionInfo.getMessage,
+              stackTraceOpt = stackTraceOpt)
+          case None =>
+            error
+        }
+      }
+    } catch {
+      case _: StatusRuntimeException => errors

Review Comment:
   No? The current expectation on this additional rpc call is best effort and at most once



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310937318


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {

Review Comment:
   Yes, the ids of different errors are unique. I updated the comment below:
   ```
     // (Required)
     // Provide a list of error ids to get. The ids of different errors are unique.
     repeated string error_ids = 3;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1681298268

   @hvanhovell Please take a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1300893112


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   and actually I think these configurations have to placed under `org.apache.spark.sql.connect.config.Connect`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1309562656


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   @heyihong if you want to set the conf without getting yelled at by SQLConf, use `onCreate(SQLConf.register)` when you are building the conf.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310937715


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {
+    // Unique identifier for this specific exception instance.
+    string error_id = 1;
+
+    // Fully qualified names of the exception class and its parental classes.
+    repeated string error_type_hierarchy = 2;
+
+    // Detailed message of the exception.
+    string message = 3;
+
+    // StackTrace of the exception.
+    repeated StackTraceElementInfo stack_trace = 4;
+  }
+
+  // List of matched exceptions.
+  repeated ExceptionInfo exception_infos = 1;

Review Comment:
   I changed it to a map so it does not have order guarantee now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310930760


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;

Review Comment:
   For Python Client, it only needs to fetch one error. For Scala Client, it needs to fetch multiple errors (including causes).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312520702


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// FetchErrorDetailsRequest defines criteria for retrieving cached exceptions.
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is true.
+  optional bool stacktrace_in_message = 4;
+}
+
+// FetchErrorDetailsResponse returns list of exceptions matching the given error ids.
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // Fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // Name of the method containing the execution point.
+    string method_name = 2;
+
+    // Name of the file containing the execution point.
+    string file_name = 3;
+
+    // Line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo represents the serialized form of an exception.
+  message ExceptionInfo {
+    // Fully qualified names of the exception class and its parent classes.
+    repeated string error_type_hierarchy = 2;
+
+    // Detailed message of the exception.
+    string message = 3;
+
+    // StackTrace of the exception. It will be set
+    // if the SQLConf spark.sql.connect.serverStacktrace.enabled is true.
+    repeated StackTraceElement stack_trace = 4;
+
+    // The id of the cause error.

Review Comment:
   `The id of the error that caused this one`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310921357


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -40,7 +42,15 @@ import org.apache.spark.sql.connect.service.SparkConnectService
 import org.apache.spark.sql.internal.SQLConf
 
 private[connect] object ErrorUtils extends Logging {
-  private def allClasses(cl: Class[_]): Seq[Class[_]] = {
+  private[connect] def allThrowables(t: Throwable): List[Throwable] = {
+    if (t == null) {
+      List.empty
+    } else {
+      t :: allThrowables(t.getCause)

Review Comment:
   I took a look but didn't find a proper use case. Do we actually use suppresed throwables in Spark? If not, I think we can suppress them for now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329300029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(

Review Comment:
   I will remove it then...
   
   The issue here is that unlike Python or other non-jvm client, they can directly fallback to use message and classes in ErrorInfo if the RPC fails. For scala client, doing such fallback means losing cause exceptions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1725111995

   @grundprinzip Here is the pr for server-side changes + python client change: https://github.com/apache/spark/pull/42987


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330148437


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -3292,39 +3292,49 @@ def test_stop_session(self):
             self.spark.conf.get("some.conf")
         self._check_no_active_session_error(e.exception)
 
-    def test_error_stack_trace(self):
-        with self.sql_conf({"spark.sql.pyspark.jvmStacktrace.enabled": True}):

Review Comment:
   Why do you need two different configs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330596345


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +214,21 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =

Review Comment:
   +1, for now, it may be better to have a flag to disable this feature in case something is not working.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330611551


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   I can follow up on this in a separate pr. This seems to be done for better performance but I am not sure if it is really necessary



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,7 +76,59 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false): FetchErrorDetailsResponse = {
+    val errors =
+      traverseCauses(st).take(MAX_ERROR_CHAIN_LENGTH).zipWithIndex.map { case (error, idx) =>
+        val builder = FetchErrorDetailsResponse.Error
+          .newBuilder()
+          .setMessage(error.getMessage)
+          .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+        if (error.getCause != null) {
+          builder.setCauseIdx(idx + 1)

Review Comment:
   Good catch!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329209246


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString

Review Comment:
   ```suggestion
         // Generate a new unique key for this exception.
         val errorId = UUID.randomUUID().toString
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329203077


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +46,11 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  val errorIdToError =

Review Comment:
   doc? what does it collect? Is it root errors or any error? What is the relationship between errors in the cache?



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(

Review Comment:
   Doc?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329221618


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   is this expected? Should we log debug something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329223475


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),

Review Comment:
   What's the best way to manage this? Can we add some tests that make sure that we don't forget get list exceptions here? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329349965


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
 
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
+  private def exceptionInfoToThrowable(exceptionInfo: ExceptionInfo): Option[Throwable] = {
+    val classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala
 
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
+    if (classHierarchy.isEmpty) {
+      return None
+    }
 
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
+    val constructor =
+      classHierarchy
+        .flatMap(errorFactory.get)
+        .headOption
+        .getOrElse((message: String, cause: Option[Throwable]) =>
+          new SparkException(
+            s"Exception in server ${classHierarchy.head}: ${message}",
+            cause.orNull))
+
+    val causeOpt = if (exceptionInfo.hasCause) {
+      exceptionInfoToThrowable(exceptionInfo.getCause)
+    } else {
+      None
+    }

Review Comment:
   It fallbacks to the abbreviated FetchErrorDetailsResponse. 
   
   ```
     for (enrichErrorEnabled <- Seq(false, true)) {
       test(s"throw SparkException with large cause exception - $enrichErrorEnabled") {
         withSQLConf("spark.sql.connect.enrichError.enabled" -> enrichErrorEnabled.toString) {
           val session = spark
           import session.implicits._
   
           val throwException =
             udf((_: String) => throw new SparkException("test" * 10000))
   
           val ex = intercept[SparkException] {
             Seq("1").toDS.withColumn("udf_val", throwException($"value")).collect()
           }
   
           assert(ex.getCause.isInstanceOf[SparkException])
           if (enrichErrorEnabled) {
             assert(ex.getCause.getMessage.contains("test" * 10000))
           } else {
             assert(ex.getCause.getMessage.startsWith("test"))
           }
         }
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329353558


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -291,15 +307,16 @@ object SparkConnectService extends Logging {
   }
 
   // Simple builder for creating the cache of Sessions.
-  private def cacheBuilder(cacheSize: Int, timeoutSeconds: Int): CacheBuilder[Object, Object] = {
+  private[service] def cacheBuilder(

Review Comment:
   Yes, SessionHolder needs to access cacheBuilder



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329640613


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   > Should we log debug something?
   
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329968772


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -26,47 +26,131 @@ import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse.ExceptionInfo
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter handles the conversion of StatusRuntimeExceptions into Spark exceptions.
+ * It does so by utilizing the ErrorInfo defined in error_details.proto and making an additional
+ * FetchErrorDetails RPC call to retrieve the full error message and optionally the server-side
+ * stacktrace.
+ *
+ * If the FetchErrorDetails RPC call succeeds, the exceptions will be constructed based on the
+ * response. If the RPC call fails, the exception will be constructed based on the ErrorInfo. If
+ * the ErrorInfo is missing, the exception will be constructed based on the StatusRuntimeException
+ * itself.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new WrappedCloseableIterator[T] {
 
       override def innerIterator: Iterator[T] = iter
 
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  /**
+   * fetchEnrichedError fetches enriched errors with full exception message and optionally
+   * stacktrace by issuing an additional RPC call to fetch error details. The RPC call is
+   * best-effort at-most-once.
+   */
+  private def fetchEnrichedError(
+      info: ErrorInfo,
+      sessionId: String,
+      userContext: UserContext): Option[Throwable] = {
+    val errorId = info.getMetadataOrDefault("errorId", null)
+    if (errorId == null) {
+      return None

Review Comment:
   This is in the client library and overall the library seems to a bit lack of logging support at the moment. For improving the logging can we do it as a follow up ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1708778584

   @hvanhovell @grundprinzip Please take another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1301913064


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -2843,6 +2843,21 @@ object SQLConf {
       // show full stacktrace in tests but hide in production by default.
       .createWithDefault(Utils.isTesting)
 
+  val SPARK_ENRICH_ERROR_ENABLED =
+    buildConf("spark.sql.spark.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("3.5.0")

Review Comment:
   > Would have to be `4.0.0` - I think it's too late for 3.5.0.
   
   Can `3.5.0` have this pr with `spark.sql.spark.enrichError.enabled` disabled by default?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1322651945


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -90,33 +218,34 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
-
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
-
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
-
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
-
-    if (errorInfoOpt.isEmpty) {
-      return fallbackEx
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
+
+  private case class Error(
+      classHierarchy: Array[String],
+      message: String,
+      stackTraceOpt: Option[Array[StackTraceElement]],
+      causeOpt: Option[Error])
+
+  private def errorToThrowable(error: Error): Option[Throwable] = {
+    if (error.classHierarchy.isEmpty) {
+      return None
     }
+    val constructor =
+      error.classHierarchy

Review Comment:
   Can you make this an iterator. So you don't have to traverse the entire error hierarchy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1686998896

   > Would be great if we have the user-facing exception (and stacktrace) example at the PR description.
   
   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42377:
URL: https://github.com/apache/spark/pull/42377#issuecomment-1687405967

   I think this needs a signoff from either @grundprinzip or @hvanhovell 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329214010


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(

Review Comment:
   This logic gets unnecessarily complex because the implementation now has to deal with two cases, injected into the response and explicit RPC. If we keep it just as the explicit RPC, the extra branches are not needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329216034


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when

Review Comment:
   what does this mean? Given the below structured data, why do you need to repeat the stack trace in the message itself?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329306123


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   For exception cases (unlike logical plan, they won't be very deep), both recursive / iterative schemas seem to work well and recursive schemas seem to be simpler to use and extend. Especially we may consider supporting suppressed exceptions. 
   
   I don't have strong opinions on either approach and a list of exceptions is actually my original proposal... I can change it back...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329328782


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          isTruncated = false,
+          serverStackTraceEnabled =
+            sessionHolder.session.conf.get(Connect.CONNECT_SERVER_STACKTRACE_ENABLED),
+          pySparkJVMStackTraceEnabled =
+            sessionHolder.session.conf.get(SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),

Review Comment:
   Connect.CONNECT_SERVER_STACKTRACE_ENABLED is a more language agnostic conf for returning stack trace.
   
   SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED is python client specific 



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectFetchErrorDetailsHandler.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.utils.ErrorUtils
+import org.apache.spark.sql.internal.SQLConf
+
+class SparkConnectFetchErrorDetailsHandler(
+    responseObserver: StreamObserver[proto.FetchErrorDetailsResponse]) {
+
+  def handle(v: proto.FetchErrorDetailsRequest): Unit = {
+    val sessionHolder =
+      SparkConnectService
+        .getOrCreateIsolatedSession(v.getUserContext.getUserId, v.getSessionId)
+
+    val response = Option(sessionHolder.errorIdToError.getIfPresent(v.getErrorId))
+      .map { error =>
+        ErrorUtils.throwableToFetchErrorDetailsResponse(
+          st = error,
+          isTruncated = false,
+          serverStackTraceEnabled =
+            sessionHolder.session.conf.get(Connect.CONNECT_SERVER_STACKTRACE_ENABLED),
+          pySparkJVMStackTraceEnabled =
+            sessionHolder.session.conf.get(SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED),

Review Comment:
   Connect.CONNECT_SERVER_STACKTRACE_ENABLED is a language-agnostic conf for returning stack trace.
   
   SQLConf.PYSPARK_JVM_STACKTRACE_ENABLED is python client specific 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330217235


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -213,4 +205,19 @@ object Connect {
     .version("3.5.0")
     .intConf
     .createWithDefault(200)
+
+  val CONNECT_ENRICH_ERROR_ENABLED =
+    buildStaticConf("spark.connect.enrichError.enabled")
+      .doc("When true, it enriches errors with full exception messages on the client side.")
+      .version("4.0.0")
+      .booleanConf
+      .createWithDefault(true)
+
+  val CONNECT_SERVER_STACKTRACE_ENABLED =

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330611551


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -66,14 +137,26 @@ private[connect] object ErrorUtils extends Logging {
         "classes",
         JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
 
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+    if (sessionHolder.session.conf.get(Connect.CONNECT_ENRICH_ERROR_ENABLED)) {
+      // Generate a new unique key for this exception.
+      val errorId = UUID.randomUUID().toString
+
+      errorInfo.putMetadata("errorId", errorId)
+
+      sessionHolder.errorIdToError
+        .put(errorId, st)
     }
 
+    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))

Review Comment:
   I can follow up on this in a separate pr. This seems to be done for better performance



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330656751


##########
connector/connect/server/src/test/scala/org/apache/spark/sql/connect/service/FetchErrorDetailsHandlerSuite.scala:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.service
+
+import java.util.UUID
+
+import scala.concurrent.Promise
+import scala.concurrent.duration._
+
+import io.grpc.stub.StreamObserver
+
+import org.apache.spark.connect.proto
+import org.apache.spark.connect.proto.FetchErrorDetailsResponse
+import org.apache.spark.sql.connect.ResourceHelper
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.ThreadUtils
+
+private class FetchErrorDetailsResponseObserver(p: Promise[FetchErrorDetailsResponse])
+    extends StreamObserver[FetchErrorDetailsResponse] {
+  override def onNext(v: FetchErrorDetailsResponse): Unit = p.success(v)
+  override def onError(throwable: Throwable): Unit = throw throwable
+  override def onCompleted(): Unit = {}
+}
+
+class FetchErrorDetailsHandlerSuite extends SharedSparkSession with ResourceHelper {

Review Comment:
   Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329282029


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))

Review Comment:
   In case the FetchErrorDetails RPC failed, fallback to an abbreviated FetchErrorDetailsResponse is consistent with the existing control. Otherwise, we may have to throw an Exception or fallback to an exception without cause support (the same code may have different behaviors due to the FetchErrorDetails RPC failure).
   
   This is for dealing with failures cases. If we find it unnecessary, we can remove this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329283591


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -3292,39 +3292,49 @@ def test_stop_session(self):
             self.spark.conf.get("some.conf")
         self._check_no_active_session_error(e.exception)
 
-    def test_error_stack_trace(self):
-        with self.sql_conf({"spark.sql.pyspark.jvmStacktrace.enabled": True}):

Review Comment:
   `spark.sql.pyspark.jvmStacktrace.enabled` is only used by spark connect python client and the option still works with this change



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -265,6 +271,12 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
 
 object SessionHolder {
 
+  // The maxmimum number of errors in the cache.
+  private val ERROR_CACHE_SIZE = 20
+
+  // The maximum time for an error to stay in the cache.
+  private val ERROR_CACHE_TIMEOUT_SEC = 60
+

Review Comment:
   20 distinct exceptions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329283591


##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -3292,39 +3292,49 @@ def test_stop_session(self):
             self.spark.conf.get("some.conf")
         self._check_no_active_session_error(e.exception)
 
-    def test_error_stack_trace(self):
-        with self.sql_conf({"spark.sql.pyspark.jvmStacktrace.enabled": True}):

Review Comment:
   `spark.sql.pyspark.jvmStacktrace.enabled` only used by spark connect python client and the option still works



##########
python/pyspark/sql/tests/connect/test_connect_basic.py:
##########
@@ -3292,39 +3292,49 @@ def test_stop_session(self):
             self.spark.conf.get("some.conf")
         self._check_no_active_session_error(e.exception)
 
-    def test_error_stack_trace(self):
-        with self.sql_conf({"spark.sql.pyspark.jvmStacktrace.enabled": True}):

Review Comment:
   `spark.sql.pyspark.jvmStacktrace.enabled` is only used by spark connect python client and the option still works



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1329349965


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -93,33 +177,44 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
       new SparkArrayIndexOutOfBoundsException(message)),
     errorConstructor[DateTimeException]((message, _) => new SparkDateTimeException(message)),
     errorConstructor((message, cause) => new SparkRuntimeException(message, cause)),
-    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)))
+    errorConstructor((message, cause) => new SparkUpgradeException(message, cause)),
+    errorConstructor((message, cause) => new SparkException(message, cause.orNull)))
 
-  private def errorInfoToThrowable(info: ErrorInfo, message: String): Option[Throwable] = {
-    val classes =
-      mapper.readValue(info.getMetadataOrDefault("classes", "[]"), classOf[Array[String]])
+  private def exceptionInfoToThrowable(exceptionInfo: ExceptionInfo): Option[Throwable] = {
+    val classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala
 
-    classes
-      .find(errorFactory.contains)
-      .map { cls =>
-        val constructor = errorFactory.get(cls).get
-        constructor(message, None)
-      }
-  }
-
-  private def toThrowable(ex: StatusRuntimeException): Throwable = {
-    val status = StatusProto.fromThrowable(ex)
-
-    val fallbackEx = new SparkException(ex.toString, ex.getCause)
+    if (classHierarchy.isEmpty) {
+      return None
+    }
 
-    val errorInfoOpt = status.getDetailsList.asScala
-      .find(_.is(classOf[ErrorInfo]))
+    val constructor =
+      classHierarchy
+        .flatMap(errorFactory.get)
+        .headOption
+        .getOrElse((message: String, cause: Option[Throwable]) =>
+          new SparkException(
+            s"Exception in server ${classHierarchy.head}: ${message}",
+            cause.orNull))
+
+    val causeOpt = if (exceptionInfo.hasCause) {
+      exceptionInfoToThrowable(exceptionInfo.getCause)
+    } else {
+      None
+    }

Review Comment:
   It fallbacks to the abbreviated FetchErrorDetailsResponse. The below test case covers the fallback case.
   
   ```
     for (enrichErrorEnabled <- Seq(false, true)) {
       test(s"throw SparkException with large cause exception - $enrichErrorEnabled") {
         withSQLConf("spark.sql.connect.enrichError.enabled" -> enrichErrorEnabled.toString) {
           val session = spark
           import session.implicits._
   
           val throwException =
             udf((_: String) => throw new SparkException("test" * 10000))
   
           val ex = intercept[SparkException] {
             Seq("1").toDS.withColumn("udf_val", throwException($"value")).collect()
           }
   
           assert(ex.getCause.isInstanceOf[SparkException])
           if (enrichErrorEnabled) {
             assert(ex.getCause.getMessage.contains("test" * 10000))
           } else {
             assert(ex.getCause.getMessage.startsWith("test"))
           }
         }
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330158282


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +79,103 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  // The maximum length of the error chain.
+  private val MAX_ERROR_CHAIN_LENGTH = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 2048
+
+  /**
+   * Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+   * @param st
+   *   the Throwable to be converted
+   * @param serverStackTraceEnabled
+   *   whether to return the server stack trace.
+   * @param pySparkJVMStackTraceEnabled
+   *   whether to return the server stack trace for Python client.
+   * @param stackTraceInMessage
+   *   whether to include the server stack trace in the message.
+   * @return
+   *   FetchErrorDetailsResponse
+   */
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {

Review Comment:
   I still don't understand the need for this flag, and I don't concur with this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] grundprinzip commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330156472


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:
##########
@@ -57,28 +69,105 @@ private[connect] object ErrorUtils extends Logging {
     classes.toSeq
   }
 
-  private def buildStatusFromThrowable(st: Throwable, stackTraceEnabled: Boolean): RPCStatus = {
+  private def serializeClasses(t: Throwable): String = {
+    JsonMethods.compact(JsonMethods.render(allClasses(t.getClass).map(_.getName)))
+  }
+
+  private[connect] val NUM_ERRORS_LIMIT = 5
+
+  // We can get full exception messages and optionally stacktrace by
+  // a separate RPC call if enrichErrorEnabled is true. So imposing a smaller
+  // limit to reduce the probability of hitting the 8KB header limit.
+  private val MAX_MESSAGE_SIZE = 512
+
+  // Convert Throwable to a protobuf message FetchErrorDetailsResponse.
+  // Truncate error messages by default.
+  private[connect] def throwableToFetchErrorDetailsResponse(
+      st: Throwable,
+      isTruncated: Boolean = true,
+      serverStackTraceEnabled: Boolean = false,
+      pySparkJVMStackTraceEnabled: Boolean = false,
+      stackTraceInMessage: Boolean = false): FetchErrorDetailsResponse = {
+    val errorChain = traverseCauses(st).take(NUM_ERRORS_LIMIT)
+
+    var lastErrorOpt: Option[ExceptionInfo] = None
+
+    errorChain.reverse.map { case error =>
+      val builder = ExceptionInfo
+        .newBuilder()
+        .setMessage(error.getMessage)
+        .addAllErrorTypeHierarchy(ErrorUtils.allClasses(error.getClass).map(_.getName).asJava)
+
+      if (isTruncated) {
+        builder.setMessage(SparkConnectService.extractErrorMessage(error, MAX_MESSAGE_SIZE))
+      } else {
+        if (stackTraceInMessage) {
+          if (serverStackTraceEnabled || pySparkJVMStackTraceEnabled) {
+            builder.setMessage(
+              s"${error.getMessage}\n\n" +
+                s"JVM stacktrace:\n${ExceptionUtils.getStackTrace(error)}")
+          }
+        } else {
+          if (serverStackTraceEnabled) {
+            builder.addAllStackTrace(
+              error.getStackTrace
+                .map { stackTraceElement =>
+                  FetchErrorDetailsResponse.StackTraceElement
+                    .newBuilder()
+                    .setDeclaringClass(stackTraceElement.getClassName)
+                    .setMethodName(stackTraceElement.getMethodName)
+                    .setFileName(stackTraceElement.getFileName)
+                    .setLineNumber(stackTraceElement.getLineNumber)
+                    .build()
+                }
+                .toIterable
+                .asJava)
+          }
+        }
+      }
+
+      lastErrorOpt.foreach(builder.setCause(_))
+      lastErrorOpt = Some(builder.build())
+    }
+
+    val responseBuilder = FetchErrorDetailsResponse.newBuilder()
+
+    lastErrorOpt.foreach(responseBuilder.setExceptionInfo(_))
+
+    responseBuilder.build()
+  }
+
+  private def buildStatusFromThrowable(
+      st: Throwable,
+      enrichErrorEnabled: Boolean,
+      userId: String,
+      sessionId: String): RPCStatus = {
     val errorInfo = ErrorInfo
       .newBuilder()
       .setReason(st.getClass.getName)
       .setDomain("org.apache.spark")
-      .putMetadata(
-        "classes",
-        JsonMethods.compact(JsonMethods.render(allClasses(st.getClass).map(_.getName))))
-
-    lazy val stackTrace = Option(ExceptionUtils.getStackTrace(st))
-    val withStackTrace = if (stackTraceEnabled && stackTrace.nonEmpty) {
-      val maxSize = SparkEnv.get.conf.get(Connect.CONNECT_JVM_STACK_TRACE_MAX_SIZE)
-      errorInfo.putMetadata("stackTrace", StringUtils.abbreviate(stackTrace.get, maxSize))
-    } else {
-      errorInfo
+      .putMetadata("classes", serializeClasses(st))
+
+    if (enrichErrorEnabled) {
+      val errorId = UUID.randomUUID().toString
+
+      // The errorId of the exception.
+      errorInfo.putMetadata("errorId", errorId)
+
+      SparkConnectService
+        .getOrCreateIsolatedSession(userId, sessionId)
+        .errorIdToError
+        .put(errorId, st)
     }
 
     RPCStatus
       .newBuilder()
       .setCode(RPCCode.INTERNAL_VALUE)
-      .addDetails(ProtoAny.pack(withStackTrace.build()))
-      .setMessage(SparkConnectService.extractErrorMessage(st))
+      .addAllDetails(
+        Seq(errorInfo.build(), throwableToFetchErrorDetailsResponse(st))
+          .map(ProtoAny.pack(_))
+          .asJava)
+      .setMessage(SparkConnectService.extractErrorMessage(st, MAX_MESSAGE_SIZE))

Review Comment:
   ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1330505500


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,67 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+message FetchErrorDetailsRequest {
+
+  // (Required)
+  // The session_id specifies a Spark session for a user identified by user_context.user_id.
+  // The id should be a UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`.
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // The id of the error.
+  string error_id = 3;
+
+  // Specifies whether to include the stacktrace in the error message when
+  // the SQLConf spark.sql.connect.serverStacktrace.enabled is set to true.
+  optional bool stacktrace_in_message = 4;
+}
+
+message FetchErrorDetailsResponse {
+
+  message StackTraceElement {
+
+    // The fully qualified name of the class containing the execution point.
+    string declaring_class = 1;
+
+    // The name of the method containing the execution point.
+    string method_name = 2;
+
+    // The name of the file containing the execution point.
+    string file_name = 3;
+
+    // The line number of the source line containing the execution point.
+    int32 line_number = 4;
+  }
+
+  // ExceptionInfo defines the schema for representing exceptions. This choice
+  // employs a recursive schema rather than an iterative one, as recursive schemas
+  // tend to be more straightforward to work with, and exceptions typically do not
+  // exhibit deep nesting.

Review Comment:
   You can suppress multiple. The only thing I want to prevent is that protocol stops us from adding them. If we have a list, and use indexes to traverse it we should be good.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell closed pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell closed pull request #42377: [SPARK-44622][SQL][CONNECT] Implement FetchErrorDetails RPC
URL: https://github.com/apache/spark/pull/42377


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1323025628


##########
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##########
@@ -44,6 +45,53 @@ import org.apache.spark.sql.types._
 
 class ClientE2ETestSuite extends RemoteSparkSession with SQLHelper with PrivateMethodTester {
 
+  test("cause exceptions should work without error enrichment") {
+    withSQLConf("spark.sql.connect.enrichError.enabled" -> "false") {
+      val ex = intercept[SparkUpgradeException] {
+        spark
+          .sql("""
+            |select from_json(
+            |  '{"d": "02-29"}',
+            |  'd date',
+            |  map('dateFormat', 'MM-dd'))
+            |""".stripMargin)
+          .collect()
+      }
+      assert(ex.getCause.isInstanceOf[DateTimeException])
+    }
+  }
+
+  test("throw SparkException with large cause exception") {
+    val session = spark
+    import session.implicits._
+
+    val throwException =
+      udf((_: String) => throw new SparkException("test" * 10000))
+
+    val ex = intercept[SparkException] {
+      Seq("1").toDS.withColumn("udf_val", throwException($"value")).collect()
+    }
+
+    assert(ex.getCause.isInstanceOf[SparkException])
+    assert(ex.getCause.getMessage.contains("test" * 10000))

Review Comment:
   A message that includes big query plans?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310928148


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;
+}
+
+// GetErrorInfoResponse returns list of exceptions matching the given error ids.
+message GetErrorInfoResponse {
+
+  message StackTraceElementInfo {

Review Comment:
   Returning StackTraceElement in a structured way is good for Scala client to interpret. But may not be useful for clients implemented in other languages. I will add a field in the request to make returning structure `StackTraceElement` optional so that for Python Client, it can directly get the message with JVM stacktrace



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310930760


##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;

Review Comment:
   For Python Client, it only needs to fetch one error. For Scala Client, it needs to fetch multiple errors (including causes). So making this field repeated may the semantic of this RPC simpler



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310938778


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -201,6 +201,20 @@ class SparkConnectService(debug: Boolean) extends AsyncService with BindableServ
         sessionId = request.getSessionId)
   }
 
+  override def getErrorInfo(
+      request: proto.GetErrorInfoRequest,
+      responseObserver: StreamObserver[proto.GetErrorInfoResponse]): Unit = {
+    try {
+      new SparkConnectGetErrorInfoHandler(responseObserver).handle(request)
+    } catch {
+      ErrorUtils.handleError(
+        "getErrorInfo",
+        observer = responseObserver,
+        userId = request.getUserContext.getUserId,
+        sessionId = request.getSessionId)

Review Comment:
   The idea is to make the ErrorInfo self-contained for issuing FetchErrorDetails request. Otherwise, it may make the implementation on the client side a bit more complicate since each FetchErrorDetails call needs to figure out the corresponding session id and user id



##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala:
##########
@@ -201,6 +201,20 @@ class SparkConnectService(debug: Boolean) extends AsyncService with BindableServ
         sessionId = request.getSessionId)
   }
 
+  override def getErrorInfo(
+      request: proto.GetErrorInfoRequest,
+      responseObserver: StreamObserver[proto.GetErrorInfoResponse]): Unit = {
+    try {
+      new SparkConnectGetErrorInfoHandler(responseObserver).handle(request)
+    } catch {
+      ErrorUtils.handleError(
+        "getErrorInfo",
+        observer = responseObserver,
+        userId = request.getUserContext.getUserId,
+        sessionId = request.getSessionId)

Review Comment:
   The idea is to make the ErrorInfo self-contained for issuing the follow up FetchErrorDetails request. Otherwise, it may make the implementation on the client side a bit more complicate since each FetchErrorDetails call needs to figure out the corresponding session id and user id



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1310971269


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -45,6 +45,9 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
   private val executions: ConcurrentMap[String, ExecuteHolder] =
     new ConcurrentHashMap[String, ExecuteHolder]()
 
+  val errorIdToError =

Review Comment:
   Making them configurable may not be necessary... I added doc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] heyihong commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "heyihong (via GitHub)" <gi...@apache.org>.
heyihong commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1311420355


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -64,6 +69,94 @@ private[client] object GrpcExceptionConverter extends JsonUtils {
     }
   }
 
+  private def enrichError(errors: Array[Error], info: ErrorInfo): Array[Error] = {
+    val sessionId = info.getMetadataOrDefault("sessionId", null)
+    val userId = info.getMetadataOrDefault("userId", null)

Review Comment:
   @hvanhovell Would like to know your thoughts on this



##########
connector/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -778,6 +778,62 @@ message ReleaseExecuteResponse {
   optional string operation_id = 2;
 }
 
+// GetErrorInfoRequest defines criteria for retrieving cached exceptions.
+message GetErrorInfoRequest {
+
+  // (Required)
+  //
+  // The session_id specifies a spark session for a user id (which is specified
+  // by user_context.user_id).
+  // The id should be an UUID string of the format `00112233-4455-6677-8899-aabbccddeeff`
+  string session_id = 1;
+
+  // User context
+  UserContext user_context = 2;
+
+  // (Required)
+  // Provide a list of error ids to get.
+  repeated string error_ids = 3;

Review Comment:
   @hvanhovell Would like to know your thoughts on this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312522446


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {

Review Comment:
   Does is need to be a List, or can we also generalize to Seq?



##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {
+    exceptionInfos.get(errorId) match {
+      case Some(exceptionInfo) =>
+        val stackTraceOpt =
+          if (exceptionInfo.getStackTraceList.isEmpty) None
+          else {
+            Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+              new StackTraceElement(
+                stackTraceElement.getDeclaringClass,
+                stackTraceElement.getMethodName,
+                stackTraceElement.getFileName,
+                stackTraceElement.getLineNumber)
+            })
+          }
+        val causeErrors = if (exceptionInfo.hasCauseErrorId) {
+          exceptionInfosToErrors(exceptionInfo.getCauseErrorId, exceptionInfos)
+        } else {
+          List.empty

Review Comment:
   Nil?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] hvanhovell commented on a diff in pull request #42377: [SPARK-44622][SQL][CONNECT] Implement error enrichment and setting server-side stacktrace

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #42377:
URL: https://github.com/apache/spark/pull/42377#discussion_r1312522517


##########
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:
##########
@@ -19,51 +19,184 @@ package org.apache.spark.sql.connect.client
 import java.time.DateTimeException
 
 import scala.collection.JavaConverters._
+import scala.collection.mutable
 import scala.reflect.ClassTag
 
 import com.google.rpc.ErrorInfo
 import io.grpc.StatusRuntimeException
 import io.grpc.protobuf.StatusProto
+import org.json4s.DefaultFormats
+import org.json4s.jackson.JsonMethods
 
 import org.apache.spark.{SparkArithmeticException, SparkArrayIndexOutOfBoundsException, SparkDateTimeException, SparkException, SparkIllegalArgumentException, SparkNumberFormatException, SparkRuntimeException, SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.connect.proto.{FetchErrorDetailsRequest, FetchErrorDetailsResponse, UserContext}
+import org.apache.spark.connect.proto.SparkConnectServiceGrpc.SparkConnectServiceBlockingStub
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.analysis.{NamespaceAlreadyExistsException, NoSuchDatabaseException, NoSuchTableException, TableAlreadyExistsException, TempTableAlreadyExistsException}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.catalyst.trees.Origin
-import org.apache.spark.util.JsonUtils
 
-private[client] object GrpcExceptionConverter extends JsonUtils {
-  def convert[T](f: => T): T = {
+/**
+ * GrpcExceptionConverter converts StatusRuntimeException to Spark exceptions.
+ * @param grpcStub
+ *   grpcStub for fetching error details from server.
+ */
+private[client] class GrpcExceptionConverter(grpcStub: SparkConnectServiceBlockingStub) {
+  import GrpcExceptionConverter._
+
+  def convert[T](sessionId: String, userContext: UserContext)(f: => T): T = {
     try {
       f
     } catch {
       case e: StatusRuntimeException =>
-        throw toThrowable(e)
+        throw toThrowable(e, sessionId, userContext)
     }
   }
 
-  def convertIterator[T](iter: CloseableIterator[T]): CloseableIterator[T] = {
+  def convertIterator[T](
+      sessionId: String,
+      userContext: UserContext,
+      iter: CloseableIterator[T]): CloseableIterator[T] = {
     new CloseableIterator[T] {
       override def hasNext: Boolean = {
-        convert {
+        convert(sessionId, userContext) {
           iter.hasNext
         }
       }
 
       override def next(): T = {
-        convert {
+        convert(sessionId, userContext) {
           iter.next()
         }
       }
 
       override def close(): Unit = {
-        convert {
+        convert(sessionId, userContext) {
           iter.close()
         }
       }
     }
   }
 
+  private def exceptionInfosToErrors(
+    errorId: String,
+    exceptionInfos: mutable.Map[String, FetchErrorDetailsResponse.ExceptionInfo]
+  ): List[Error] = {
+    exceptionInfos.get(errorId) match {
+      case Some(exceptionInfo) =>
+        val stackTraceOpt =
+          if (exceptionInfo.getStackTraceList.isEmpty) None
+          else {
+            Some(exceptionInfo.getStackTraceList.asScala.toArray.map { stackTraceElement =>
+              new StackTraceElement(
+                stackTraceElement.getDeclaringClass,
+                stackTraceElement.getMethodName,
+                stackTraceElement.getFileName,
+                stackTraceElement.getLineNumber)
+            })
+          }
+        val causeErrors = if (exceptionInfo.hasCauseErrorId) {
+          exceptionInfosToErrors(exceptionInfo.getCauseErrorId, exceptionInfos)
+        } else {
+          List.empty
+        }
+        Error(
+          classHierarchy = exceptionInfo.getErrorTypeHierarchyList.asScala.toArray,
+          message = exceptionInfo.getMessage,
+          stackTraceOpt = stackTraceOpt
+        ) +: causeErrors
+      case None =>
+        List.empty

Review Comment:
   Nil?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org