You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/08/25 15:34:19 UTC
(spark) branch branch-4.3 updated: [SPARK-58931][SQL] Reject negative randstr length during analysis
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new cfa026cbc0a7 [SPARK-58931][SQL] Reject negative randstr length during analysis
cfa026cbc0a7 is described below
commit cfa026cbc0a7250f78d2e5d13ff0ec6c7cd555f2
Author: Uros <22...@users.noreply.github.com>
AuthorDate: Tue Aug 25 15:33:27 2026 +0000
[SPARK-58931][SQL] Reject negative randstr length during analysis
### What changes were proposed in this pull request?
`randstr(length[, seed])` requires a non-negative `length`. Today that guard lives only on the execution paths - `RandStr.lengthInteger()`, called from `evalInternal` (interpreted) and `doGenCode` (codegen), so a negative constant `length` is not rejected until the query executes.
This PR moves the guard into Catalyst analysis. `RandStr.checkInputDataTypes()` already requires `length` to be a foldable integer; once those checks pass, it now evaluates the (constant) `length` and, if it is negative, returns a failed `TypeCheckResult` - `DataTypeMismatch("VALUE_OUT_OF_RANGE")` - rather than throwing. A `null` `length` is left untouched (`randstr(NULL, 0)` remains valid and returns an empty string). Because `checkInputDataTypes()` runs during analysis, `randstr` wi [...]
This follows the precedent of other expressions that validate a foldable constant during analysis - `TimeBucket` (`datetimeExpressions.scala`) and `RegExpReplace` (`regexpExpressions.scala`) - which `eval()` the constant and return `DataTypeMismatch("VALUE_OUT_OF_RANGE")`.
### Why are the changes needed?
`randstr`'s `length` must be a foldable constant, which `checkInputDataTypes()` already enforces, so a negative `length` can be detected during analysis rather than only at execution. Performing the check in `checkInputDataTypes()`:
- rejects an invalid `randstr(-1, ...)` during analysis (fail-fast), rather than only once the expression is evaluated at execution;
- surfaces the failure as an `AnalysisException` with a `QueryContext`, like every other `randstr` input check, instead of a `SparkRuntimeException` escaping from analysis;
- keeps the validation with the rest of `randstr`'s input checking.
### Does this PR introduce _any_ user-facing change?
Yes. `randstr` with a negative constant `length` now fails during analysis instead of at execution, and the error class changes.
- Before: analysis succeeds; the query fails at execution with a `SparkRuntimeException`, error class `INVALID_PARAMETER_VALUE.LENGTH`.
- After: the query fails during analysis with an `AnalysisException`, error class `DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE`.
Because the failure now happens during analysis, it affects queries that never reach execution:
- `EXPLAIN SELECT randstr(-1, 0)` previously printed a plan; it now fails.
- `spark.sql("SELECT randstr(-1, 0)")` now throws immediately (before any action), so the returned `DataFrame`'s `schema` is unreachable.
- A `randstr(-1, ...)` in a branch the optimizer would have pruned away now fails rather than being eliminated.
Failing fast on an invalid constant `length` is the intent of this change. Queries with a valid (non-negative or `NULL`) `length` are unaffected.
### How was this patch tested?
- Added a negative-length case to `DataFrameFunctionsSuite`'s `test("randstr function")` asserting that `df.select(randstr(lit(-1), lit(0)))` fails during analysis (on `select` alone, without an action) with `DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE`.
- The golden SQL test `random.sql` covers `SELECT randstr(-1, 0)`; regenerated both `analyzer-results/random.sql.out` and `results/random.sql.out` and reviewed the diff:
```
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z random.sql"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #58200 from uros-b/randstr-neg-length-analysis.
Authored-by: Uros <22...@users.noreply.github.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit d01d0205f250acffe242d01eab6b225deeb346f6)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
.../catalyst/expressions/randomExpressions.scala | 20 ++++++++++++++++----
.../sql-tests/analyzer-results/random.sql.out | 19 ++++++++++++++++++-
.../resources/sql-tests/results/random.sql.out | 22 +++++++++++++++-------
.../apache/spark/sql/DataFrameFunctionsSuite.scala | 17 +++++++++++++++++
4 files changed, 66 insertions(+), 12 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala
index b52d09fc9c70..4009e292916d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala
@@ -21,7 +21,7 @@ import org.apache.spark.SparkException
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.{TypeCheckResult, UnresolvedSeed}
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
-import org.apache.spark.sql.catalyst.expressions.ExpectsInputTypes.{toSQLExpr, toSQLId}
+import org.apache.spark.sql.catalyst.expressions.ExpectsInputTypes.{toSQLExpr, toSQLId, toSQLValue}
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, CodeGenerator, ExprCode, FalseLiteral}
import org.apache.spark.sql.catalyst.expressions.codegen.Block._
import org.apache.spark.sql.catalyst.trees.{BinaryLike, TernaryLike, UnaryLike}
@@ -345,13 +345,13 @@ object Uniform {
usage = """
_FUNC_(length[, seed]) - Returns a string of the specified length whose characters are chosen
uniformly at random from the following pool of characters: 0-9, a-z, A-Z. The random seed is
- optional. The string length must be a constant two-byte or four-byte integer (SMALLINT or INT,
- respectively).
+ optional. The string length must be a non-negative constant two-byte or four-byte integer
+ (SMALLINT or INT, respectively).
""",
arguments = """
Arguments:
* length - The length of the random string to generate.
- An expression that evaluates to an integer. Must be a constant.
+ An expression that evaluates to a non-negative integer. Must be a constant.
* seed - The seed used to produce reproducible random results.
An expression that evaluates to an integer or long. Must be a constant.
""",
@@ -425,6 +425,18 @@ case class RandStr(
"inputExpr" -> toSQLExpr(expr)))
}
}
+ if (result == TypeCheckResult.TypeCheckSuccess) {
+ val lengthValue = length.eval()
+ // randstr(NULL, 0) is valid (treated as 0), so only reject a negative length.
+ if (lengthValue != null && lengthValue.asInstanceOf[Int] < 0) {
+ result = DataTypeMismatch(
+ errorSubClass = "VALUE_OUT_OF_RANGE",
+ messageParameters = Map(
+ "exprName" -> toSQLId("length"),
+ "valueRange" -> s"[0, ${Int.MaxValue}]",
+ "currentValue" -> toSQLValue(lengthValue, IntegerType)))
+ }
+ }
result
}
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/random.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/random.sql.out
index 96a4b2ec91c7..0f19b148db17 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/random.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/random.sql.out
@@ -709,7 +709,24 @@ org.apache.spark.sql.AnalysisException
-- !query
SELECT randstr(-1, 0) AS result
-- !query analysis
-[Analyzer test output redacted due to nondeterminism]
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "currentValue" : "-1",
+ "exprName" : "`length`",
+ "sqlExpr" : "\"randstr(-1, 0)\"",
+ "valueRange" : "[0, 2147483647]"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 21,
+ "fragment" : "randstr(-1, 0)"
+ } ]
+}
-- !query
diff --git a/sql/core/src/test/resources/sql-tests/results/random.sql.out b/sql/core/src/test/resources/sql-tests/results/random.sql.out
index 049d134003cd..5439683037b7 100644
--- a/sql/core/src/test/resources/sql-tests/results/random.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/random.sql.out
@@ -857,15 +857,23 @@ SELECT randstr(-1, 0) AS result
-- !query schema
struct<>
-- !query output
-org.apache.spark.SparkRuntimeException
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
{
- "errorClass" : "INVALID_PARAMETER_VALUE.LENGTH",
- "sqlState" : "22023",
+ "errorClass" : "DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE",
+ "sqlState" : "42K09",
"messageParameters" : {
- "functionName" : "`randstr`",
- "length" : "-1",
- "parameter" : "`length`"
- }
+ "currentValue" : "-1",
+ "exprName" : "`length`",
+ "sqlExpr" : "\"randstr(-1, 0)\"",
+ "valueRange" : "[0, 2147483647]"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 21,
+ "fragment" : "randstr(-1, 0)"
+ } ]
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index f86613bf6684..e32ea5e66f4b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -499,6 +499,23 @@ class DataFrameFunctionsSuite extends SharedSparkSession {
callSitePattern = "",
startIndex = 0,
stopIndex = 0))
+ expr = randstr(lit(-1), lit(0))
+ checkError(
+ intercept[AnalysisException](df.select(expr)),
+ condition = "DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE",
+ parameters = Map(
+ "sqlExpr" -> "\"randstr(-1, 0)\"",
+ "exprName" -> "`length`",
+ "valueRange" -> "[0, 2147483647]",
+ "currentValue" -> "-1"),
+ context = ExpectedContext(
+ contextType = QueryContextType.DataFrame,
+ fragment = "randstr",
+ objectType = "",
+ objectName = "",
+ callSitePattern = "",
+ startIndex = 0,
+ stopIndex = 0))
}
test("uniform function") {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org