You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2026/02/23 21:35:39 UTC
[PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
parthchandra opened a new pull request, #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580
## Which issue does this PR close?
Closes parts of https://github.com/apache/datafusion-comet/issues/551
Closes https://github.com/apache/datafusion-comet/issues/2215
Closes https://github.com/apache/datafusion-comet/issues/3375
## Rationale for this change
With Spark 4.0 (And Spark 3.5 with Ansi mode), Spark produces ansi compliant error messages that have an error code and in many cases include the original SQL query. When we encounter errors in native code, Comet throws a SparkException or CometNativeException that do not conform to the expected error reporting standard.
## What changes are included in this PR?
This PR introduces a framework to report ansi compliant error messages from native code.
Summary of error propagation -
1. Spark-side query context serialization : For every serialized expression and aggregate expression, a unique `expr_id` is generated. If the expression's origin carries a `QueryContext` (SQL text, line, column, object name), it is extracted and attached to the protobuf. This is done for both `Expr` and `AggExpr`.
2. Native planner (planner.rs): The PhysicalPlanner now holds a `QueryContextMap`. When planning `Expr` and `AggExpr` nodes, if `expr_id` and `query_context` are present, the context is registered in the map. When creating physical expressions for Cast, CheckOverflow, ListExtract, SumDecimal, AvgDecimal, and arithmetic binary expressions, the relevant QueryContext is looked up and passed to the constructor.
3. Native errors : The `SparkError` enum is extended with new variants will all the Spark ANSI errors (from https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala). A new `SparkErrorWithContext` type wraps a `SparkError` with a `QueryContext`. All affected expression implementations look up the context and produce a `SparkErrorWithContext` when available.
The `SparkError` implementation also has new `to_json()` and `exception_class()` methods for JNI serialization.
4. JNI boundary (errors.rs -> CometQueryExecutionException): The `throw_exception` function now checks for `SparkErrorWithContext` or `SparkError` and throws `CometQueryExecutionException`. `CometQueryExecutionException` carries the entire `SparkErrorWithContext` as a JSON message. On the Scala side, `CometExecIterator` catches this exception and calls `SparkErrorConverter.convertToSparkException()` to convert to the appropriate Spark exception. If the JSON message contained the `QueryContext`, the exception will contain the query, otherwise it will not.
5. There are two version specific implementations -one for Spark 3.x (fallback to generic SparkException) and one for Spark 4.0 (calls the exact QueryExecutionErrors.* methods).
Notes: Not all expressions have been updated. All the expressions that failed unit tests as a result of incorrect error messages have been updated. ( Cast, CheckOverflow, ListExtract, SumDecimal, AvgDecimal, and binary arithmetic expressions). Binary arithmetic expressions are now represented as `CheckedBinaryExpr` which also includes the query context.
Most errors in `QueryExecutionErrors` are reproduced as is in the native side. However some errors like `INTERVAL_ARITHMETIC_OVERFLOW` have a version with a user suggestion and one without a user suggestion. In such cases there are two variants in the native side.
## How are these changes tested?
New unit tests. Failing tests listed in https://github.com/apache/datafusion-comet/issues/551, https://github.com/apache/datafusion-comet/issues/2215, https://github.com/apache/datafusion-comet/issues/3375
_This PR was produced with the generous assistance of Claude Code_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#issuecomment-4000099692
Thank you Andy. Will wait for https://github.com/apache/datafusion-comet/pull/3559 before rebasing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra merged PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2843476469
##########
spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.shims
+
+import org.apache.spark.QueryContext
+import org.apache.spark.SparkException
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Spark 4.0-specific implementation for converting error types to proper Spark exceptions.
+ */
+trait ShimSparkErrorConverter {
+
+ /**
+ * Convert error type string and parameters to appropriate Spark exception. Version-specific
+ * implementations call the correct QueryExecutionErrors.* methods.
+ *
+ * @param errorType
+ * The error type from JSON (e.g., "DivideByZero")
+ * @param errorClass
+ * The Spark error class (e.g., "DIVIDE_BY_ZERO")
+ * @param params
+ * Error parameters from JSON
+ * @param context
+ * QueryContext array with SQL text and position information
+ * @param summary
+ * Formatted summary string showing error location
+ * @return
+ * Throwable (specific exception type from QueryExecutionErrors), or None if unknown
+ */
+ def convertErrorType(
+ errorType: String,
+ _errorClass: String,
+ params: Map[String, Any],
+ context: Array[QueryContext],
+ _summary: String): Option[Throwable] = {
+
+ errorType match {
+
+ case "DivideByZero" =>
+ Some(QueryExecutionErrors.divideByZeroError(context.headOption.orNull))
+
+ case "RemainderByZero" =>
+ // SPARK 4.0 REMOVED remainderByZeroError so we use generic arithmetic exception
+ Some(
+ new SparkException(
+ errorClass = "REMAINDER_BY_ZERO",
+ messageParameters = params.map { case (k, v) => (k, v.toString) },
+ cause = null))
+
+ case "IntervalDividedByZero" =>
+ Some(QueryExecutionErrors.intervalDividedByZeroError(context.headOption.orNull))
+
+ case "BinaryArithmeticOverflow" =>
+ Some(
+ QueryExecutionErrors.binaryArithmeticCauseOverflowError(
+ params("value1").toString.toShort,
Review Comment:
Could this overflow if the value is larger than a short?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "coderfender (via GitHub)" <gi...@apache.org>.
coderfender commented on PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#issuecomment-3947627272
Thank you @parthchandra . This is awesome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2874866659
##########
native/org/apache/spark/sql/errors/QueryExecutionErrors.scala:
##########
@@ -0,0 +1,2718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import java.io.{FileNotFoundException, IOException}
+import java.lang.reflect.InvocationTargetException
+import java.net.{URISyntaxException, URL}
+import java.time.DateTimeException
+import java.util.concurrent.TimeoutException
+
+import com.fasterxml.jackson.core.{JsonParser, JsonToken}
+import org.apache.hadoop.fs.{FileAlreadyExistsException, FileStatus, Path}
+import org.apache.hadoop.fs.permission.FsPermission
+import org.codehaus.commons.compiler.{CompileException, InternalCompilerException}
+
+import org.apache.spark._
+import org.apache.spark.launcher.SparkLauncher
+import org.apache.spark.memory.SparkOutOfMemoryError
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.ScalaReflection.Schema
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator
+import org.apache.spark.sql.catalyst.catalog.{CatalogDatabase, CatalogTable}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.plans.JoinType
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.statsEstimation.ValueInterval
+import org.apache.spark.sql.catalyst.trees.{Origin, SQLQueryContext, TreeNode}
+import org.apache.spark.sql.catalyst.util.{sideBySide, BadRecordException, DateTimeUtils, FailFastMode}
+import org.apache.spark.sql.connector.catalog.{CatalogNotFoundException, Table, TableProvider}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.GLOBAL_TEMP_DATABASE
+import org.apache.spark.sql.streaming.OutputMode
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.array.ByteArrayMethods
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.{CircularBuffer, Utils}
+
+/**
+ * Object for grouping error messages from (most) exceptions thrown during query execution.
+ * This does not include exceptions thrown during the eager execution of commands, which are
+ * grouped into [[QueryCompilationErrors]].
+ */
+private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionErrors {
Review Comment:
This Scala class is in the native module. Should this be here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2875359356
##########
native/spark-expr/src/agg_funcs/avg_decimal.rs:
##########
@@ -240,37 +311,44 @@ impl Accumulator for AvgDecimalAccumulator {
// of the computation
return Ok(());
}
-
let values = &values[0];
let data = values.as_primitive::<Decimal128Type>();
-
self.is_empty = self.is_empty && values.len() == values.null_count();
-
if values.null_count() == 0 {
for i in 0..data.len() {
- self.update_single(data, i);
+ self.update_single(data, i)?;
}
} else {
for i in 0..data.len() {
if data.is_null(i) {
continue;
}
- self.update_single(data, i);
+ self.update_single(data, i)?;
}
}
Ok(())
}
fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+ let partial_sums = states[0].as_primitive::<Decimal128Type>();
+ let partial_counts = states[1].as_primitive::<Int64Type>();
+
+ // Update is_empty: if any partial state has data, we're not empty
+ if self.is_empty {
+ self.is_empty = partial_counts.len() == partial_counts.null_count();
+ }
+
// counts are summed
- self.count += sum(states[1].as_primitive::<Int64Type>()).unwrap_or_default();
+ self.count += sum(partial_counts).unwrap_or_default();
// sums are summed
- if let Some(x) = sum(states[0].as_primitive::<Decimal128Type>()) {
+ if let Some(x) = sum(partial_sums) {
let v = self.sum.get_or_insert(0);
let (result, overflowed) = v.overflowing_add(x);
- if overflowed {
- // Set to None if overflow happens
+
+ if overflowed || !is_valid_decimal_precision(result, self.sum_precision) {
Review Comment:
This looks like a change in functionality? Why is the `is_valid_decimal_precision` check now needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2875447206
##########
common/src/main/java/org/apache/comet/exceptions/CometQueryExecutionException.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.exceptions;
+
+import org.apache.comet.CometNativeException;
+
+/**
+ * Exception thrown from Comet native execution containing JSON-encoded error information. The
+ * message contains a JSON object with the following structure:
+ *
+ * <pre>
+ * {
+ * "errorType": "DivideByZero",
+ * "errorClass": "DIVIDE_BY_ZERO",
+ * "params": { ... },
+ * "context": { "sqlText": "...", "startOffset": 0, "stopOffset": 10 },
+ * "hint": "Use `try_divide` to tolerate divisor being 0"
+ * }
+ * </pre>
+ *
+ * CometExecIterator parses this JSON and converts it to the appropriate Spark exception by calling
+ * the corresponding QueryExecutionErrors.* method.
+ */
+public final class CometQueryExecutionException extends CometNativeException {
+
+ /**
+ * Creates a new CometQueryExecutionException with a JSON-encoded error message.
+ *
+ * @param jsonMessage JSON string containing error information
+ */
+ public CometQueryExecutionException(String jsonMessage) {
+ super(jsonMessage);
+ }
+
+ /**
+ * Returns true if the message appears to be JSON-formatted. This is used to distinguish between
+ * JSON-encoded errors and legacy error messages.
+ *
+ * @return true if message starts with '{' and ends with '}'
+ */
+ public boolean isJsonMessage() {
+ String msg = getMessage();
+ return msg != null && msg.trim().startsWith("{") && msg.trim().endsWith("}");
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#issuecomment-3987587018
Also fixed another failing test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2875335880
##########
common/src/main/java/org/apache/comet/exceptions/CometQueryExecutionException.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet.exceptions;
+
+import org.apache.comet.CometNativeException;
+
+/**
+ * Exception thrown from Comet native execution containing JSON-encoded error information. The
+ * message contains a JSON object with the following structure:
+ *
+ * <pre>
+ * {
+ * "errorType": "DivideByZero",
+ * "errorClass": "DIVIDE_BY_ZERO",
+ * "params": { ... },
+ * "context": { "sqlText": "...", "startOffset": 0, "stopOffset": 10 },
+ * "hint": "Use `try_divide` to tolerate divisor being 0"
+ * }
+ * </pre>
+ *
+ * CometExecIterator parses this JSON and converts it to the appropriate Spark exception by calling
+ * the corresponding QueryExecutionErrors.* method.
+ */
+public final class CometQueryExecutionException extends CometNativeException {
+
+ /**
+ * Creates a new CometQueryExecutionException with a JSON-encoded error message.
+ *
+ * @param jsonMessage JSON string containing error information
+ */
+ public CometQueryExecutionException(String jsonMessage) {
+ super(jsonMessage);
+ }
+
+ /**
+ * Returns true if the message appears to be JSON-formatted. This is used to distinguish between
+ * JSON-encoded errors and legacy error messages.
+ *
+ * @return true if message starts with '{' and ends with '}'
+ */
+ public boolean isJsonMessage() {
+ String msg = getMessage();
+ return msg != null && msg.trim().startsWith("{") && msg.trim().endsWith("}");
Review Comment:
nit: `trim` is called twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2843561540
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1526,20 +1553,11 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
if (CometSparkSessionExtensions.isSpark40Plus) {
// for Spark 4 we expect to sparkException carries the message
assert(sparkMessage.contains("SQLSTATE"))
- if (sparkMessage.startsWith("[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION]")) {
- assert(
- sparkMessage.replace(".WITH_SUGGESTION] ", "]").startsWith(cometMessage))
- } else if (cometMessage.startsWith("[CAST_INVALID_INPUT]") || cometMessage
- .startsWith("[CAST_OVERFLOW]")) {
- assert(
- sparkMessage.startsWith(
- cometMessage
- .replace(
- "If necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error.",
- "")))
- } else {
- assert(sparkMessage.startsWith(cometMessage))
- }
+ // we compare a subset of the error message. Comet grabs the query
+ // context eagerly so it displays the call site at the
+ // line of code where the cast method was called, whereas spark grabs the context
+ // lazily and displays the call site at the line of code where the error is checked.
+ assert(sparkMessage.startsWith(cometMessage.substring(0, 40)))
Review Comment:
I used 40 as an arbitrary number to get enough of the error message. There is no special significance.
##########
spark/src/main/spark-4.0/org/apache/spark/sql/comet/shims/ShimSparkErrorConverter.scala:
##########
@@ -0,0 +1,280 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet.shims
+
+import org.apache.spark.QueryContext
+import org.apache.spark.SparkException
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+/**
+ * Spark 4.0-specific implementation for converting error types to proper Spark exceptions.
+ */
+trait ShimSparkErrorConverter {
+
+ /**
+ * Convert error type string and parameters to appropriate Spark exception. Version-specific
+ * implementations call the correct QueryExecutionErrors.* methods.
+ *
+ * @param errorType
+ * The error type from JSON (e.g., "DivideByZero")
+ * @param errorClass
+ * The Spark error class (e.g., "DIVIDE_BY_ZERO")
+ * @param params
+ * Error parameters from JSON
+ * @param context
+ * QueryContext array with SQL text and position information
+ * @param summary
+ * Formatted summary string showing error location
+ * @return
+ * Throwable (specific exception type from QueryExecutionErrors), or None if unknown
+ */
+ def convertErrorType(
+ errorType: String,
+ _errorClass: String,
+ params: Map[String, Any],
+ context: Array[QueryContext],
+ _summary: String): Option[Throwable] = {
+
+ errorType match {
+
+ case "DivideByZero" =>
+ Some(QueryExecutionErrors.divideByZeroError(context.headOption.orNull))
+
+ case "RemainderByZero" =>
+ // SPARK 4.0 REMOVED remainderByZeroError so we use generic arithmetic exception
+ Some(
+ new SparkException(
+ errorClass = "REMAINDER_BY_ZERO",
+ messageParameters = params.map { case (k, v) => (k, v.toString) },
+ cause = null))
+
+ case "IntervalDividedByZero" =>
+ Some(QueryExecutionErrors.intervalDividedByZeroError(context.headOption.orNull))
+
+ case "BinaryArithmeticOverflow" =>
+ Some(
+ QueryExecutionErrors.binaryArithmeticCauseOverflowError(
+ params("value1").toString.toShort,
Review Comment:
I believe that should not happen. The original Spark exception has these values declared as `short`.
##########
native/spark-expr/src/query_context.rs:
##########
@@ -0,0 +1,360 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Query execution context for error reporting
+//!
+//! This module provides QueryContext which mirrors Spark's SQLQueryContext
+//! for providing SQL text, line/position information, and error location
+//! pointers in exception messages.
+
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+/// Based on Spark's SQLQueryContext for error reporting.
+///
+/// Contains information about where an error occurred in a SQL query,
+/// including the full SQL text, line/column positions, and object context.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
+pub struct QueryContext {
+ /// Full SQL query text
+ #[serde(rename = "sqlText")]
+ pub sql_text: Arc<String>,
+
+ /// Start offset in SQL text (0-based, character index)
+ #[serde(rename = "startIndex")]
+ pub start_index: i32,
+
+ /// Stop offset in SQL text (0-based, character index, inclusive)
+ #[serde(rename = "stopIndex")]
+ pub stop_index: i32,
+
+ /// Object type (e.g., "VIEW", "Project", "Filter")
+ #[serde(rename = "objectType", skip_serializing_if = "Option::is_none")]
+ pub object_type: Option<String>,
+
+ /// Object name (e.g., view name, column name)
+ #[serde(rename = "objectName", skip_serializing_if = "Option::is_none")]
+ pub object_name: Option<String>,
+
+ /// Line number in SQL query (1-based)
+ pub line: i32,
+
+ /// Column position within the line (0-based)
+ #[serde(rename = "startPosition")]
+ pub start_position: i32,
+}
+
+impl QueryContext {
+ #[allow(clippy::too_many_arguments)]
+ pub fn new(
+ sql_text: String,
+ start_index: i32,
+ stop_index: i32,
+ object_type: Option<String>,
+ object_name: Option<String>,
+ line: i32,
+ start_position: i32,
+ ) -> Self {
+ Self {
+ sql_text: Arc::new(sql_text),
+ start_index,
+ stop_index,
+ object_type,
+ object_name,
+ line,
+ start_position,
+ }
+ }
+
+ /// Generate a summary string showing SQL fragment with error location.
+ /// (From SQLQueryContext.summary)
+ ///
+ /// Format example:
+ /// ```text
+ /// == SQL of VIEW v1 (line 1, position 8) ==
+ /// SELECT a/b FROM t
+ /// ^^^
+ /// ```
+ pub fn format_summary(&self) -> String {
+ let start_idx = self.start_index.max(0) as usize;
+ let stop_idx = (self.stop_index + 1).max(0) as usize;
+
+ // Extract the problematic fragment
+ let fragment = if start_idx < self.sql_text.len() && stop_idx <= self.sql_text.len() {
+ &self.sql_text[start_idx..stop_idx]
Review Comment:
Good catch. Fixed to use char index. Added a unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#issuecomment-3947582111
@coderfender, fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2843471451
##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -1526,20 +1553,11 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
if (CometSparkSessionExtensions.isSpark40Plus) {
// for Spark 4 we expect to sparkException carries the message
assert(sparkMessage.contains("SQLSTATE"))
- if (sparkMessage.startsWith("[NUMERIC_VALUE_OUT_OF_RANGE.WITH_SUGGESTION]")) {
- assert(
- sparkMessage.replace(".WITH_SUGGESTION] ", "]").startsWith(cometMessage))
- } else if (cometMessage.startsWith("[CAST_INVALID_INPUT]") || cometMessage
- .startsWith("[CAST_OVERFLOW]")) {
- assert(
- sparkMessage.startsWith(
- cometMessage
- .replace(
- "If necessary set \"spark.sql.ansi.enabled\" to \"false\" to bypass this error.",
- "")))
- } else {
- assert(sparkMessage.startsWith(cometMessage))
- }
+ // we compare a subset of the error message. Comet grabs the query
+ // context eagerly so it displays the call site at the
+ // line of code where the cast method was called, whereas spark grabs the context
+ // lazily and displays the call site at the line of code where the error is checked.
+ assert(sparkMessage.startsWith(cometMessage.substring(0, 40)))
Review Comment:
is 40 just an arbitrary number to get a decent sized chunk of the error message or does it have a specific meaning in the context of this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2843494632
##########
native/spark-expr/src/query_context.rs:
##########
@@ -0,0 +1,360 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Query execution context for error reporting
+//!
+//! This module provides QueryContext which mirrors Spark's SQLQueryContext
+//! for providing SQL text, line/position information, and error location
+//! pointers in exception messages.
+
+use serde::{Deserialize, Serialize};
+use std::sync::Arc;
+
+/// Based on Spark's SQLQueryContext for error reporting.
+///
+/// Contains information about where an error occurred in a SQL query,
+/// including the full SQL text, line/column positions, and object context.
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
+pub struct QueryContext {
+ /// Full SQL query text
+ #[serde(rename = "sqlText")]
+ pub sql_text: Arc<String>,
+
+ /// Start offset in SQL text (0-based, character index)
+ #[serde(rename = "startIndex")]
+ pub start_index: i32,
+
+ /// Stop offset in SQL text (0-based, character index, inclusive)
+ #[serde(rename = "stopIndex")]
+ pub stop_index: i32,
+
+ /// Object type (e.g., "VIEW", "Project", "Filter")
+ #[serde(rename = "objectType", skip_serializing_if = "Option::is_none")]
+ pub object_type: Option<String>,
+
+ /// Object name (e.g., view name, column name)
+ #[serde(rename = "objectName", skip_serializing_if = "Option::is_none")]
+ pub object_name: Option<String>,
+
+ /// Line number in SQL query (1-based)
+ pub line: i32,
+
+ /// Column position within the line (0-based)
+ #[serde(rename = "startPosition")]
+ pub start_position: i32,
+}
+
+impl QueryContext {
+ #[allow(clippy::too_many_arguments)]
+ pub fn new(
+ sql_text: String,
+ start_index: i32,
+ stop_index: i32,
+ object_type: Option<String>,
+ object_name: Option<String>,
+ line: i32,
+ start_position: i32,
+ ) -> Self {
+ Self {
+ sql_text: Arc::new(sql_text),
+ start_index,
+ stop_index,
+ object_type,
+ object_name,
+ line,
+ start_position,
+ }
+ }
+
+ /// Generate a summary string showing SQL fragment with error location.
+ /// (From SQLQueryContext.summary)
+ ///
+ /// Format example:
+ /// ```text
+ /// == SQL of VIEW v1 (line 1, position 8) ==
+ /// SELECT a/b FROM t
+ /// ^^^
+ /// ```
+ pub fn format_summary(&self) -> String {
+ let start_idx = self.start_index.max(0) as usize;
+ let stop_idx = (self.stop_index + 1).max(0) as usize;
+
+ // Extract the problematic fragment
+ let fragment = if start_idx < self.sql_text.len() && stop_idx <= self.sql_text.len() {
+ &self.sql_text[start_idx..stop_idx]
Review Comment:
The earlier docs say that `start_idx` is a character index, but it is being used as a byte index here, I think. Perhaps you could tests for non-ASCII cases, if that makes sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#issuecomment-3948368310
Changed to draft to figure out backward compatibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2875198895
##########
native/org/apache/spark/sql/errors/QueryExecutionErrors.scala:
##########
@@ -0,0 +1,2718 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.errors
+
+import java.io.{FileNotFoundException, IOException}
+import java.lang.reflect.InvocationTargetException
+import java.net.{URISyntaxException, URL}
+import java.time.DateTimeException
+import java.util.concurrent.TimeoutException
+
+import com.fasterxml.jackson.core.{JsonParser, JsonToken}
+import org.apache.hadoop.fs.{FileAlreadyExistsException, FileStatus, Path}
+import org.apache.hadoop.fs.permission.FsPermission
+import org.codehaus.commons.compiler.{CompileException, InternalCompilerException}
+
+import org.apache.spark._
+import org.apache.spark.launcher.SparkLauncher
+import org.apache.spark.memory.SparkOutOfMemoryError
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.ScalaReflection.Schema
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.UnresolvedGenerator
+import org.apache.spark.sql.catalyst.catalog.{CatalogDatabase, CatalogTable}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParseException
+import org.apache.spark.sql.catalyst.plans.JoinType
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.statsEstimation.ValueInterval
+import org.apache.spark.sql.catalyst.trees.{Origin, SQLQueryContext, TreeNode}
+import org.apache.spark.sql.catalyst.util.{sideBySide, BadRecordException, DateTimeUtils, FailFastMode}
+import org.apache.spark.sql.connector.catalog.{CatalogNotFoundException, Table, TableProvider}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.StaticSQLConf.GLOBAL_TEMP_DATABASE
+import org.apache.spark.sql.streaming.OutputMode
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.array.ByteArrayMethods
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.{CircularBuffer, Utils}
+
+/**
+ * Object for grouping error messages from (most) exceptions thrown during query execution.
+ * This does not include exceptions thrown during the eager execution of commands, which are
+ * grouped into [[QueryCompilationErrors]].
+ */
+private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionErrors {
Review Comment:
No idea how this got there. This is a spark class! Removed.
Thanks for catching this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: [ANSI] Ansi sql error messages [datafusion-comet]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #3580:
URL: https://github.com/apache/datafusion-comet/pull/3580#discussion_r2875444499
##########
native/spark-expr/src/agg_funcs/avg_decimal.rs:
##########
@@ -240,37 +311,44 @@ impl Accumulator for AvgDecimalAccumulator {
// of the computation
return Ok(());
}
-
let values = &values[0];
let data = values.as_primitive::<Decimal128Type>();
-
self.is_empty = self.is_empty && values.len() == values.null_count();
-
if values.null_count() == 0 {
for i in 0..data.len() {
- self.update_single(data, i);
+ self.update_single(data, i)?;
}
} else {
for i in 0..data.len() {
if data.is_null(i) {
continue;
}
- self.update_single(data, i);
+ self.update_single(data, i)?;
}
}
Ok(())
}
fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+ let partial_sums = states[0].as_primitive::<Decimal128Type>();
+ let partial_counts = states[1].as_primitive::<Int64Type>();
+
+ // Update is_empty: if any partial state has data, we're not empty
+ if self.is_empty {
+ self.is_empty = partial_counts.len() == partial_counts.null_count();
+ }
+
// counts are summed
- self.count += sum(states[1].as_primitive::<Int64Type>()).unwrap_or_default();
+ self.count += sum(partial_counts).unwrap_or_default();
// sums are summed
- if let Some(x) = sum(states[0].as_primitive::<Decimal128Type>()) {
+ if let Some(x) = sum(partial_sums) {
let v = self.sum.get_or_insert(0);
let (result, overflowed) = v.overflowing_add(x);
- if overflowed {
- // Set to None if overflow happens
+
+ if overflowed || !is_valid_decimal_precision(result, self.sum_precision) {
Review Comment:
This is a tricky one. `overflowing_add` checks for integer overflow but decimal can overflow if the precision is exceeded so we need to check for that as well. This is caught by the `DataFrameAggregateSuite.checkAggResultsForDecimalOverflow` test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org