You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/09/10 13:10:41 UTC
(spark) branch branch-4.x updated: [SPARK-58713][SQL] Fix integer overflow in overlay() position arithmetic
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e60e19c5bdec [SPARK-58713][SQL] Fix integer overflow in overlay() position arithmetic
e60e19c5bdec is described below
commit e60e19c5bdec52bc0374ced8b851c0eefeab5f54
Author: Sepuri Sai Krishna <sa...@gmail.com>
AuthorDate: Thu Sep 10 13:10:20 2026 +0000
[SPARK-58713][SQL] Fix integer overflow in overlay() position arithmetic
### What changes were proposed in this pull request?
`Overlay.calculate` computes its two slice boundaries in `int` arithmetic:
```scala
builder.append(input.substringSQL(1, pos - 1)) // (1) `pos - 1` overflows
...
builder.append(input.substringSQL(pos + length, Int.MaxValue)) // (2) `pos + length` overflows
```
The BINARY overload has the same two expressions, via `ByteArray.subStringSQL`.
This PR computes both boundaries as `long` and saturates them into the `int` range before
calling `substringSQL`, through a small private helper:
```scala
private def clamp(value: Long): Int =
math.max(Int.MinValue.toLong, math.min(Int.MaxValue.toLong, value)).toInt
```
Saturating rather than throwing keeps `overlay()` consistent with `substring()`, which already
treats out-of-range positions by clamping. Only the four boundary expressions change; the rest of
both overloads is untouched.
### Why are the changes needed?
For `pos == Int.MinValue`, `pos - 1` wraps to `Int.MaxValue`, so the leading slice becomes the
whole input instead of the empty string. For `pos` near `Int.MaxValue`, `pos + length` wraps to a
large negative value, which `substringSQL` interprets as an offset from the end of the input, so
the trailing slice becomes the whole input instead of the empty string. Either way the input is
duplicated around the replacement, silently and with no error raised.
```sql
SELECT overlay('Spark SQL' PLACING '_' FROM 2147483647 FOR 5);
-- before: 'Spark SQL_Spark SQL'
-- after: 'Spark SQL_'
SELECT overlay('Spark SQL' PLACING '_' FROM -2147483648 FOR 1);
-- before: 'Spark SQL_Spark SQL'
-- after: '_Spark SQL'
```
Both the STRING and the BINARY overload are affected, on the interpreted and the codegen path,
with default configuration.
### Does this PR introduce _any_ user-facing change?
Yes, for positions near the ends of the `int` range, which are currently wrong. Measured with
input `'Spark SQL'` and replacement `'_'`:
| pos | len | before (STRING) | before (BINARY) | after (both) |
|---|---|---|---|---|
| `Int.MaxValue` | 5 | `Spark SQL_Spark SQL` | `Spark SQL_Spark SQL` | `Spark SQL_` |
| `Int.MaxValue` | default | `Spark SQL_Spark SQ` | `Spark SQL_Spark SQL` | `Spark SQL_` |
| `Int.MaxValue - 2` | 10 | `Spark SQL_Spark SQL` | `Spark SQL_Spark SQL` | `Spark SQL_` |
| `Int.MinValue` | 1 | `Spark SQL_Spark SQL` | `Spark SQL_Spark SQL` | `_Spark SQL` |
| `Int.MinValue` | default | `Spark SQL_Spark SQL` | `Spark SQL_Spark SQL` | `_Spark SQL` |
No change for any in-range position. The existing test cases covering positions 0, 2, 6, 7, 10 and
-10 pass unmodified.
Note the STRING and BINARY columns already disagree on the second row. That is a symptom of a
separate clamping problem in `ByteArray.subStringSQL`, fixed under SPARK-58708 (#57922). The two
are independent: the overflow addressed here happens in `Overlay.calculate` before `substringSQL`
is ever reached, so neither PR fixes the other's bug, and they can be merged in either order.
### How was this patch tested?
Five new cases added to each of the existing `overlay for string` and `overlay for byte array`
tests in `StringExpressionsSuite`, covering both overflow directions and both the two-argument
(default length) and three-argument forms. These go through `checkEvaluation`, so the interpreted
and codegen paths are both exercised.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
Closes #57931 from SEPURI-SAI-KRISHNA/SPARK-58713-overlay-overflow.
Authored-by: Sepuri Sai Krishna <sa...@gmail.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit 2ffbc96c7f84a7830d7244f7b0c0617de31fb18b)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
.../catalyst/expressions/stringExpressions.scala | 25 ++++++--
.../expressions/StringExpressionsSuite.scala | 73 ++++++++++++++++++++++
2 files changed, 94 insertions(+), 4 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
index afd894167ac9..763e5dbe8648 100755
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
@@ -1096,9 +1096,24 @@ case class StringReplace(srcExpr: Expression, searchExpr: Expression, replaceExp
object Overlay {
+ // `pos - 1` and `pos + length` overflow for positions near the ends of the `int` range,
+ // which turns an empty slice into the whole input and duplicates it around the
+ // replacement. Compute them as `long` and saturate into the `int` range instead, which
+ // is how `substring` already treats out-of-range positions.
+ private def clamp(value: Long): Int =
+ math.max(Int.MinValue.toLong, math.min(Int.MaxValue.toLong, value)).toInt
+
+ // The tail of the result starts at `pos + length`. `substringSQL` derives its end offset from
+ // `start + Int.MaxValue`, which still reaches the end of the input for every position down to
+ // `Int.MinValue + 1` but falls one character short at `Int.MinValue` itself. Both denote the
+ // same tail, the whole input, so raise the floor by one rather than reading the input length,
+ // which would cost an O(n) scan on every call to spare the one extreme position.
+ private def clampTail(value: Long): Int =
+ math.max(Int.MinValue.toLong + 1, math.min(Int.MaxValue.toLong, value)).toInt
+
def calculate(input: UTF8String, replace: UTF8String, pos: Int, len: Int): UTF8String = {
val builder = new UTF8StringBuilder
- builder.append(input.substringSQL(1, pos - 1))
+ builder.append(input.substringSQL(1, clamp(pos.toLong - 1)))
builder.append(replace)
// If you specify length, it must be a positive whole number or zero.
// Otherwise it will be ignored.
@@ -1108,7 +1123,8 @@ object Overlay {
} else {
replace.numChars
}
- builder.append(input.substringSQL(pos + length, Int.MaxValue))
+ val tail = clampTail(pos.toLong + length)
+ builder.append(input.substringSQL(tail, Int.MaxValue))
builder.build()
}
@@ -1121,8 +1137,9 @@ object Overlay {
} else {
replace.length
}
- ByteArray.concat(ByteArray.subStringSQL(input, 1, pos - 1),
- replace, ByteArray.subStringSQL(input, pos + length, Int.MaxValue))
+ val tail = clampTail(pos.toLong + length)
+ ByteArray.concat(ByteArray.subStringSQL(input, 1, clamp(pos.toLong - 1)),
+ replace, ByteArray.subStringSQL(input, tail, Int.MaxValue))
}
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
index 7c8afb7cba7c..bf2ddddd6778 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/StringExpressionsSuite.scala
@@ -853,6 +853,60 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
Literal.create(-10, IntegerType)), "__park SQL")
checkEvaluation(Overlay(Literal("Spark SQL"), Literal("__"),
Literal.create(-10, IntegerType), Literal.create(4, IntegerType)), "__rk SQL")
+ // SPARK-58713: `pos - 1` and `pos + len` must not overflow the int range. Before the
+ // fix each of these duplicated the input around the replacement.
+ checkEvaluation(Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MaxValue, IntegerType), Literal.create(5, IntegerType)), "Spark SQL_")
+ checkEvaluation(new Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MaxValue, IntegerType)), "Spark SQL_")
+ checkEvaluation(Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MaxValue - 2, IntegerType), Literal.create(10, IntegerType)),
+ "Spark SQL_")
+ checkEvaluation(Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(1, IntegerType)), "_Spark SQL")
+ checkEvaluation(new Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MinValue, IntegerType)), "_Spark SQL")
+ // SPARK-58713: with a zero length the tail starts at `Int.MinValue` itself. `substringSQL`
+ // only treats an end offset of exactly `Int.MaxValue` as the rest of the input, so without
+ // clamping the tail position this dropped the last character and disagreed with the binary
+ // overload below.
+ checkEvaluation(Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(0, IntegerType)), "_Spark SQL")
+ checkEvaluation(Overlay(Literal("Spark SQL"), Literal("_"),
+ Literal.create(Int.MinValue + 1, IntegerType), Literal.create(0, IntegerType)), "_Spark SQL")
+ // The tail position is clamped in characters, so a multi-byte input keeps every character,
+ // and an empty input has no tail to keep.
+ // scalastyle:off nonascii
+ checkEvaluation(Overlay(Literal("caf\u00e9 SQL"), Literal("_"),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(0, IntegerType)), "_caf\u00e9 SQL")
+ // scalastyle:on nonascii
+ checkEvaluation(Overlay(Literal(""), Literal("_"),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(0, IntegerType)), "_")
+ }
+
+ test("SPARK-58713: overlay position arithmetic is collation-independent") {
+ // `Overlay` builds its result from character positions and `substringSQL`, and never
+ // consults the collation - `CollationSupport` defines no collation-aware overlay - so
+ // the clamped out-of-range positions fixed above produce the same result under every
+ // collation. The mixed case of the expected values also shows nothing is case folded.
+ Seq("UTF8_BINARY", "UTF8_LCASE", "UNICODE", "UNICODE_CI").foreach { collation =>
+ val st = StringType(collation)
+ val input = Literal.create("Spark SQL", st)
+ val replace = Literal.create("_", st)
+ checkEvaluation(Overlay(input, replace,
+ Literal.create(Int.MaxValue, IntegerType), Literal.create(5, IntegerType)),
+ "Spark SQL_")
+ checkEvaluation(new Overlay(input, replace,
+ Literal.create(Int.MaxValue, IntegerType)), "Spark SQL_")
+ checkEvaluation(Overlay(input, replace,
+ Literal.create(Int.MaxValue - 2, IntegerType), Literal.create(10, IntegerType)),
+ "Spark SQL_")
+ checkEvaluation(Overlay(input, replace,
+ Literal.create(Int.MinValue, IntegerType), Literal.create(1, IntegerType)),
+ "_Spark SQL")
+ checkEvaluation(new Overlay(input, replace,
+ Literal.create(Int.MinValue, IntegerType)), "_Spark SQL")
+ }
}
test("overlay for byte array") {
@@ -891,6 +945,25 @@ class StringExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
Literal.create(-10, IntegerType)), Array[Byte](-1, -1, 2, 3, 4, 5, 6, 7, 8, 9))
checkEvaluation(Overlay(input, Literal(Array[Byte](-1, -1)), Literal.create(-10, IntegerType),
Literal.create(4, IntegerType)), Array[Byte](-1, -1, 4, 5, 6, 7, 8, 9))
+ // SPARK-58713: `pos - 1` and `pos + len` must not overflow the int range. Before the
+ // fix each of these duplicated the input around the replacement.
+ checkEvaluation(Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MaxValue, IntegerType), Literal.create(5, IntegerType)),
+ Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9, -1))
+ checkEvaluation(new Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MaxValue, IntegerType)), Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9, -1))
+ checkEvaluation(Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MaxValue - 2, IntegerType), Literal.create(10, IntegerType)),
+ Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9, -1))
+ checkEvaluation(Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(1, IntegerType)),
+ Array[Byte](-1, 1, 2, 3, 4, 5, 6, 7, 8, 9))
+ checkEvaluation(new Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MinValue, IntegerType)), Array[Byte](-1, 1, 2, 3, 4, 5, 6, 7, 8, 9))
+ // SPARK-58713: the zero-length case must agree with the string overload above.
+ checkEvaluation(Overlay(input, Literal(Array[Byte](-1)),
+ Literal.create(Int.MinValue, IntegerType), Literal.create(0, IntegerType)),
+ Array[Byte](-1, 1, 2, 3, 4, 5, 6, 7, 8, 9))
}
test("Check Overlay.checkInputDataTypes results") {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org