[jira] [Created] (SPARK-58821) sequence() raises INTERNAL_ERROR for BIGINT endpoints whose difference overflows Long

["sepuri sai krishna (Jira)" <ji...@apache.org>]

sepuri sai krishna created SPARK-58821:
------------------------------------------

             Summary: sequence() raises INTERNAL_ERROR for BIGINT endpoints whose difference overflows Long
                 Key: SPARK-58821
                 URL: https://issues.apache.org/jira/browse/SPARK-58821
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0
            Reporter: sepuri sai krishna


{{sequence(start, stop, step)}} over BIGINT raises an {{INTERNAL_ERROR}} (SQLSTATE XX000) for argument combinations that have a small, well-defined result. Reachable from plain SQL with default configuration, on both the interpreted and the codegen path.

{code:sql}
SELECT sequence(-9223372036854775808L, 9223372036854775807L, 9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
-- expected: [-9223372036854775808, -1, 9223372036854775806]

SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
-- expected: [-9223372036854775808, -4611686018427387904, 0]
{code}

The same failure occurs when the arguments are not foldable, so it is not confined to constant folding:

{code:sql}
CREATE OR REPLACE TEMP VIEW t AS
  SELECT CAST(-9223372036854775808 AS BIGINT) AS a,
         CAST(9223372036854775807 AS BIGINT) AS b,
         CAST(9223372036854775807 AS BIGINT) AS c;
SELECT sequence(a, b, c) FROM t;
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
{code}

h3. Cause

{{Sequence.sequenceLength}} computes the length in {{long}} and falls back to {{BigInt}} when that overflows:

{code:scala}
def sequenceLength(start: Long, stop: Long, step: Long): Int = {
  try {
    val delta = Math.subtractExact(stop, start)      // (1) overflows
    ...
    val len = if (stop == start) 1L else Math.addExact(1L, (delta / step))
    if (len > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
      throw QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName, len)
    }
    len.toInt
  } catch {
    case _: ArithmeticException =>
      val safeLen = BigInt(1) + (BigInt(stop) - BigInt(start)) / BigInt(step)   // (2)
      if (safeLen > ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) {
        throw QueryExecutionErrors.createArrayWithElementsExceedLimitError(prettyName, safeLen)
      }
      throw internalError("Unreachable code reached.")                          // (3)
    ...
{code}

The fallback assumes an overflow at (1) implies a result too large to allocate, so after recomputing the length exactly at (2) it only re-raises the array-limit error and otherwise treats (3) as dead code.

That assumption does not hold. {{Math.subtractExact(stop, start)}} overflows whenever {{stop - start}} exceeds the {{long}} range, which says nothing about the length: the length also depends on {{step}}. When the step is large, the exact length computed at (2) is small, no limit error is raised, and control reaches the {{internalError}} at (3) -- with the correct answer sitting in {{safeLen}}, discarded.

For the first example above, {{safeLen}} is exactly {{3}}.

h3. Scope

* BIGINT only. {{IntegralSequenceImpl}} reaches {{sequenceLength}} via {{getSequenceLength}}, which widens its arguments with {{toLong}}, so for INT/SMALLINT/TINYINT the subtraction cannot overflow. The DATE/TIMESTAMP callers pass microsecond values, whose range is far from the {{long}} limits.
* Both execution paths. The interpreted path calls {{Sequence.sequenceLength}} directly and the generated code calls the same method, so they fail identically.
* Not a recent regression. {{sequenceLength}} was introduced in its current shape by SPARK-43393 (commit {{afc4c49927cb}}, Nov 2023), which fixed a genuine correctness bug where the {{long}} length computation overflowed silently and {{sequence}} returned an empty array. That fix added the {{BigInt}} fallback and, reasonably at the time, treated the tail as unreachable -- the overflow cases it was written against all had huge lengths. The gap is that a large {{step}} makes the exact length small, so the tail is reachable after all. The method has not been modified since, so every release containing SPARK-43393 (3.3.4, 3.4.2, 3.5.1, 4.0.0 and later) is affected.

h3. Suggested fix

At (3), return {{safeLen.toInt}} instead of throwing. The value has already been proven to be at most {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}} by the check immediately above, so the narrowing conversion is safe, and the element-filling code already handles steps of this magnitude correctly -- {{sequence(-4611686018427387904L, 4611686018427387903L, 4611686018427387903L)}}, whose endpoints are close enough together to avoid the overflow, returns {{[-4611686018427387904, -1, 4611686018427387902]}} today.

Tests belong in {{CollectionExpressionsSuite}}, which exercises both the interpreted and the codegen path through {{checkEvaluation}}.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org