You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "othon2000 (via GitHub)" <gi...@apache.org> on 2026/03/10 14:10:14 UTC
[PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
othon2000 opened a new pull request, #54728:
URL: https://github.com/apache/spark/pull/54728
### What changes were proposed in this pull request?
This PR addresses two related issues in `ErrorUtils` around gRPC metadata size enforcement:
**[SPARK-55930] Byte-aware size checks (`SparkStringUtils`)**
- Add `sizeInBytes` and `abbreviateByBytes` helpers to `SparkStringUtils`
- Use `sizeInBytes` when checking `messageParameters` against the metadata size limit
- Use `abbreviateByBytes` when truncating `stackTrace` to the limit
**[SPARK-55931] Add `errorClassFallback` metadata key**
When `messageParameters` JSON exceeds `maxMetadataSize`, surface the error class via a separate
`errorClassFallback` metadata key so proxy/interceptor systems can inspect the error class
without interfering with the existing `FetchErrorDetails` client/server flow.
### Why are the changes needed?
Character-length-based checks undercount the actual byte size of gRPC metadata payloads when
multi-byte characters (e.g. CJK) are present, potentially causing `MESSAGE_SIZE_LIMIT_EXCEEDED`
errors at the transport layer: essentially an error when throwing an error, which is a debugging hell.
Additionally, when `messageParameters` are dropped due to size limits, there was no way for
observability or proxy layers to recover the error class from the metadata.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added `ErrorUtilsSuite` covering `buildStatusFromThrowable` behavior, including:
- Multi-byte character scenarios for byte-aware size checks
- `errorClassFallback` propagation when `messageParameters` exceed the limit
Extended `SparkStringUtils` with unit tests for `sizeInBytes` and `abbreviateByBytes`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Co-authored by Claude Sonnet 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #54728:
URL: https://github.com/apache/spark/pull/54728#discussion_r2915376656
##########
common/utils/src/main/scala/org/apache/spark/util/SparkStringUtils.scala:
##########
@@ -58,6 +59,25 @@ private[spark] trait SparkStringUtils {
def abbreviate(str: String, len: Int): String = abbreviate(str, "...", len)
+ def abbreviateByBytes(str: String, abbrevMarker: String, maxBytes: Int): String = {
+ if (str == null || abbrevMarker == null) {
+ null
+ } else if (sizeInBytes(str) <= maxBytes) {
+ str
+ } else {
+ val budget = maxBytes - sizeInBytes(abbrevMarker)
+ val buf = StandardCharsets.UTF_8.encode(str)
Review Comment:
Can we try to be a bit more careful with allocations. `sizeInBytes(..)` and this call both encode the string to bytes. At this point you have allocated 3 ByteBuffers. Can we bring it down to 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "othon2000 (via GitHub)" <gi...@apache.org>.
othon2000 commented on PR #54728:
URL: https://github.com/apache/spark/pull/54728#issuecomment-4037663889
The implementation still contains a bug, where we are checking individual metadata fields against the limit but not their sum. Needs to be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "othon2000 (via GitHub)" <gi...@apache.org>.
othon2000 commented on code in PR #54728:
URL: https://github.com/apache/spark/pull/54728#discussion_r2919860611
##########
common/utils/src/main/scala/org/apache/spark/util/SparkStringUtils.scala:
##########
@@ -58,6 +59,25 @@ private[spark] trait SparkStringUtils {
def abbreviate(str: String, len: Int): String = abbreviate(str, "...", len)
+ def abbreviateByBytes(str: String, abbrevMarker: String, maxBytes: Int): String = {
+ if (str == null || abbrevMarker == null) {
+ null
+ } else if (sizeInBytes(str) <= maxBytes) {
+ str
+ } else {
+ val budget = maxBytes - sizeInBytes(abbrevMarker)
+ val buf = StandardCharsets.UTF_8.encode(str)
Review Comment:
I removed one. To remove both, the implementation was quite convoluted. LMK if this is enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #54728: [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback
URL: https://github.com/apache/spark/pull/54728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #54728:
URL: https://github.com/apache/spark/pull/54728#discussion_r2915382017
##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/utils/ErrorUtilsSuite.scala:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.utils
+
+import scala.jdk.CollectionConverters._
+
+import com.google.rpc.ErrorInfo
+
+import org.apache.spark.SparkThrowable
+import org.apache.spark.sql.connect.SparkConnectTestUtils
+import org.apache.spark.sql.connect.config.Connect
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.SparkStringUtils
+
+class ErrorUtilsSuite extends SharedSparkSession {
+
+ private def makeThrowable(
+ errorClass: String,
+ messageParameters: Map[String, String]): Throwable with SparkThrowable = {
+ new Exception(s"Test error for $errorClass") with SparkThrowable {
Review Comment:
Why not use SparkException?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55930][SPARK-55931][CONNECT] Byte-aware gRPC metadata size checks and errorClassFallback [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54728:
URL: https://github.com/apache/spark/pull/54728#issuecomment-4814453089
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org