You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/22 19:39:50 UTC
(spark) branch branch-4.x updated: [SPARK-58252][SQL] Throw GROUP_BY_POS_OUT_OF_RANGE for out-of-range ordinal in pipe AGGREGATE GROUP BY
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 9d4cb165de56 [SPARK-58252][SQL] Throw GROUP_BY_POS_OUT_OF_RANGE for out-of-range ordinal in pipe AGGREGATE GROUP BY
9d4cb165de56 is described below
commit 9d4cb165de5683c00ad5d6ff8243151775eedcf8
Author: YangJie <ya...@baidu.com>
AuthorDate: Wed Jul 22 12:39:30 2026 -0700
[SPARK-58252][SQL] Throw GROUP_BY_POS_OUT_OF_RANGE for out-of-range ordinal in pipe AGGREGATE GROUP BY
### What changes were proposed in this pull request?
An out-of-range GROUP BY ordinal in the SQL pipe AGGREGATE operator threw a raw `java.lang.IndexOutOfBoundsException` instead of a proper `AnalysisException`. For example:
```sql
select 3 as x, 4 as y
|> aggregate sum(y) group by 5;
```
`resolvePipeAggregateExpressionOrdinal` resolved the ordinal with an unguarded `inputs(index - 1)`, so an out-of-range position (or `0`) surfaced the internal JVM exception.
This PR adds the same bounds guard the regular GROUP BY ordinal resolver (`resolveGroupByExpressionOrdinal`) already uses, throwing `GROUP_BY_POS_OUT_OF_RANGE` (sqlState `42805`). The `UnresolvedPipeAggregateOrdinal` node is now created with the ordinal literal's origin (via `CurrentOrigin.withOrigin`), so the error's `queryContext` points at the offending ordinal rather than the whole pipe statement, matching regular GROUP BY.
### Why are the changes needed?
A raw `IndexOutOfBoundsException` escaping the analyzer for valid-syntax SQL is an internal error leak. User-facing errors should carry an error class, sqlState, and query context. The pipe AGGREGATE path was the only pipe operator with its own GROUP BY ordinal handling and the only one missing this guard; regular `GROUP BY <ordinal>` has thrown `GROUP_BY_POS_OUT_OF_RANGE` with precise context for years.
### Does this PR introduce _any_ user-facing change?
Yes, on the error path only. For an out-of-range or non-positive ordinal in pipe AGGREGATE GROUP BY:
Before:
```
java.lang.IndexOutOfBoundsException: Index 4 out of bounds for length 2
```
After:
```
[GROUP_BY_POS_OUT_OF_RANGE] GROUP BY position 5 is not in select list (valid range is [1, 2]). SQLSTATE: 42805
```
with a query context pointing at the ordinal. Valid (in-range) ordinals are unaffected.
### How was this patch tested?
Added positive-range-adjacent negative cases to `pipe-operators.sql` (`group by 5` above range, `group by 0` non-positive) in the aggregation negative-tests section; regenerated the golden files. The regenerated goldens assert the error class, sqlState, `index`/`size` parameters, and the precise `queryContext` (fragment pointing at the ordinal). Existing positive pipe-ordinal cases (including the exact upper bound `group by x, 2, 3`) regenerated with no diff, confirming zero happy-pat [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57423 from LuciferYang/pipe-groupby-ordinal-oob.
Authored-by: YangJie <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 4462cb1dd30ac46ab27bef394c856be1e760757f)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 13 ++++--
.../spark/sql/catalyst/parser/AstBuilder.scala | 7 +++-
.../analyzer-results/pipe-operators.sql.out | 44 ++++++++++++++++++++
.../resources/sql-tests/inputs/pipe-operators.sql | 8 ++++
.../sql-tests/results/pipe-operators.sql.out | 48 ++++++++++++++++++++++
5 files changed, 115 insertions(+), 5 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index c629de82c6ff..215c859154a0 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -2252,11 +2252,18 @@ class Analyzer(
private def resolvePipeAggregateExpressionOrdinal(
expr: NamedExpression,
inputs: Seq[Attribute]): NamedExpression = expr match {
- case UnresolvedPipeAggregateOrdinal(index) =>
+ case ordinal @ UnresolvedPipeAggregateOrdinal(index) =>
// In this case, the user applied the SQL pipe aggregate operator ("|> AGGREGATE") and used
// ordinals in its GROUP BY clause. This expression then refers to the i-th attribute of the
- // child operator (one-based). Here we resolve the ordinal to the corresponding attribute.
- inputs(index - 1)
+ // child operator (one-based). Here we resolve the ordinal to the corresponding attribute, or
+ // throw GROUP_BY_POS_OUT_OF_RANGE if it is outside the range of the child's attributes.
+ withPosition(ordinal) {
+ if (index > 0 && index <= inputs.size) {
+ inputs(index - 1)
+ } else {
+ throw QueryCompilationErrors.groupByPositionRangeError(index, inputs.size)
+ }
+ }
case other =>
other
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 6b60cb9ebd3a..b18187da474a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -7919,9 +7919,12 @@ class AstBuilder extends DataTypeAstBuilder
// grouping and aggregate expressions, respectively. This will let the
// [[ResolveOrdinalInOrderByAndGroupBy]] rule detect the ordinal in the aggregate list
// and replace it with the corresponding attribute from the child operator.
- case UnresolvedOrdinal(v: Int) =>
+ case ordinal @ UnresolvedOrdinal(v: Int) =>
newGroupingExpressions += UnresolvedOrdinal(newAggregateExpressions.length + 1)
- newAggregateExpressions += UnresolvedPipeAggregateOrdinal(v)
+ // Preserve the ordinal's origin so an out-of-range error points at the position
+ // itself, not the whole pipe statement (matching regular GROUP BY).
+ newAggregateExpressions +=
+ CurrentOrigin.withOrigin(ordinal.origin) { UnresolvedPipeAggregateOrdinal(v) }
case e: Expression =>
newGroupingExpressions += e
newAggregateExpressions += UnresolvedAlias(e, None)
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out
index d136341fa32b..666c5c91f8bf 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/pipe-operators.sql.out
@@ -3492,6 +3492,50 @@ org.apache.spark.sql.catalyst.parser.ParseException
}
+-- !query
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 5
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "GROUP_BY_POS_OUT_OF_RANGE",
+ "sqlState" : "42805",
+ "messageParameters" : {
+ "index" : "5",
+ "size" : "2"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 52,
+ "fragment" : "5"
+ } ]
+}
+
+
+-- !query
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 0
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "GROUP_BY_POS_OUT_OF_RANGE",
+ "sqlState" : "42805",
+ "messageParameters" : {
+ "index" : "0",
+ "size" : "2"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 52,
+ "fragment" : "0"
+ } ]
+}
+
+
-- !query
table courseSales
|> aggregate sum(earnings) group by rollup(course, `year`)
diff --git a/sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql b/sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql
index 7347ad1385a3..916b114f17f5 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/pipe-operators.sql
@@ -1128,6 +1128,14 @@ table other
select 3 as x, 4 as y
|> aggregate group by all;
+-- The AGGREGATE GROUP BY ordinal position is out of range.
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 5;
+
+-- The AGGREGATE GROUP BY ordinal position is not positive.
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 0;
+
-- GROUP BY ROLLUP is not supported yet.
table courseSales
|> aggregate sum(earnings) group by rollup(course, `year`)
diff --git a/sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out b/sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out
index fabe74b49158..578e1478ea23 100644
--- a/sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/pipe-operators.sql.out
@@ -3109,6 +3109,54 @@ org.apache.spark.sql.catalyst.parser.ParseException
}
+-- !query
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 5
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "GROUP_BY_POS_OUT_OF_RANGE",
+ "sqlState" : "42805",
+ "messageParameters" : {
+ "index" : "5",
+ "size" : "2"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 52,
+ "fragment" : "5"
+ } ]
+}
+
+
+-- !query
+select 3 as x, 4 as y
+|> aggregate sum(y) group by 0
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "GROUP_BY_POS_OUT_OF_RANGE",
+ "sqlState" : "42805",
+ "messageParameters" : {
+ "index" : "0",
+ "size" : "2"
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 52,
+ "fragment" : "0"
+ } ]
+}
+
+
-- !query
table courseSales
|> aggregate sum(earnings) group by rollup(course, `year`)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org