You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/09/23 10:21:38 UTC
(spark) branch master updated: [SPARK-59529][SQL] Reject ASOF JOIN MATCH_CONDITION operand that references no join input
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 373c63c06ebb [SPARK-59529][SQL] Reject ASOF JOIN MATCH_CONDITION operand that references no join input
373c63c06ebb is described below
commit 373c63c06ebb25438dd80478544ee03f96113f95
Author: Luka Zdravić <lu...@gmail.com>
AuthorDate: Wed Sep 23 10:21:24 2026 +0000
[SPARK-59529][SQL] Reject ASOF JOIN MATCH_CONDITION operand that references no join input
### What changes were proposed in this pull request?
Make SQL `ASOF JOIN ... MATCH_CONDITION (a op b)` reject an operand that references no join input.
`MATCH_CONDITION` needs each operand to reference exactly one join input: one from the left, one from the right.
`AsOfJoin.operandJoinSide` picks the side from an operand's references:
- only the left input -> left
- only the right input -> right
- the wrong side or both sides -> rejected with `ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE`
- no input (a literal, or a call like `current_date()`) -> previously pinned to its syntactic side and accepted (the
bug)
This PR returns `None` for a no-reference operand, so it falls through to the same
`ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE` error already used for wrong-side references. The fixed-point
(`ResolveAsOfJoin`) and single-pass (`AsOfJoinResolver`) analyzers share this path, so both behave the same.
### Why are the changes needed?
A query that looks like an as-of join could build a real ASOF join with a constant operand and no nearest-match. Both
of these were accepted before:
```sql
SELECT * FROM (VALUES (1),(6)) l(a) ASOF JOIN (VALUES (10)) r(b)
MATCH_CONDITION (l.a >= 5);
SELECT * FROM (VALUES (DATE'2024-01-01')) l(a) ASOF JOIN (VALUES (DATE'2024-01-02')) r(b)
MATCH_CONDITION (l.a >= current_date());
```
The wrong-side case was already rejected:
```sql
SELECT * FROM (VALUES (1,2)) l(a,c) ASOF JOIN (VALUES (10)) r(b)
MATCH_CONDITION (l.a >= l.c); -- ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE
```
This PR makes the no-reference case consistent with it.
### Does this PR introduce _any_ user-facing change?
Yes, but only in the unreleased line. SQL `ASOF JOIN` is new and off by default (`spark.sql.join.asofJoin.enabled`,
added in 4.3.0). A `MATCH_CONDITION` operand that references no join input now fails analysis with `ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE` (SQLSTATE 42K0E) instead of being accepted. No released version is affected.
### How was this patch tested?
- `AsOfJoinSQLSuite`: `checkError` tests for a literal operand, `current_timestamp()`, both operands constant, and a
constant operand under the single-pass analyzer. The two tests that asserted acceptance now assert rejection.
- Golden files: added FVT-ASOF-3-024..027 in `join-asof-errors.sql`; removed the old accept case from
`join-asof-expressions.sql` and left a pointer comment; regenerated the `.sql.out` files.
- `AsOfJoinSQLSuite` and `AsOfJoinSortMergeSQLSuite` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #58853 from LukaZdravic/effort/literal-fix-spark.
Authored-by: Luka Zdravić <lu...@gmail.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
.../src/main/resources/error/error-conditions.json | 2 +-
.../sql/catalyst/analysis/ResolveAsOfJoin.scala | 4 +-
.../plans/logical/basicLogicalOperators.scala | 12 ++--
.../spark/sql/errors/QueryCompilationErrors.scala | 4 +-
.../analyzer-results/join-asof-errors.sql.out | 76 ++++++++++++++++++--
.../analyzer-results/join-asof-expressions.sql.out | 27 -------
.../sql-tests/inputs/join-asof-errors.sql | 20 ++++++
.../sql-tests/inputs/join-asof-expressions.sql | 18 +----
.../sql-tests/results/join-asof-errors.sql.out | 84 ++++++++++++++++++++--
.../results/join-asof-expressions.sql.out | 33 ---------
.../org/apache/spark/sql/AsOfJoinSQLSuite.scala | 80 +++++++++++++--------
11 files changed, 231 insertions(+), 129 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index 5ef335276bc7..eaa27251a01b 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -176,7 +176,7 @@
},
"ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE" : {
"message" : [
- "Each operand of the MATCH_CONDITION comparison must reference only columns of one join input. Got: <refs1> and <refs2>."
+ "Each operand of the MATCH_CONDITION comparison must reference exactly one join input, and the two operands must reference different inputs. Got: <operand1> and <operand2>."
],
"sqlState" : "42K0E"
},
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveAsOfJoin.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveAsOfJoin.scala
index f5f828fc6ed6..33ca926fc060 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveAsOfJoin.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveAsOfJoin.scala
@@ -188,8 +188,8 @@ private[analysis] object AsOfJoinValidation extends QueryErrorsBase {
join.failAnalysis(
errorClass = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
messageParameters = Map(
- "refs1" -> toSQLExpr(leftExpr),
- "refs2" -> toSQLExpr(rightExpr)))
+ "operand1" -> toSQLExpr(leftExpr),
+ "operand2" -> toSQLExpr(rightExpr)))
}
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
index 1b3fce928dbf..e60b8e056350 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
@@ -2936,8 +2936,8 @@ object AsOfJoin {
expr1: Expression,
operator: MatchComparisonOperator,
expr2: Expression): (Expression, Expression, MatchComparisonOperator) = {
- val expr1Side = operandJoinSide(expr1, leftSet, rightSet, syntacticIsLeft = true)
- val expr2Side = operandJoinSide(expr2, leftSet, rightSet, syntacticIsLeft = false)
+ val expr1Side = operandJoinSide(expr1, leftSet, rightSet)
+ val expr2Side = operandJoinSide(expr2, leftSet, rightSet)
(expr1Side, expr2Side) match {
case (Some(true), Some(false)) => (expr1, expr2, operator)
case (Some(false), Some(true)) => (expr2, expr1, operator.flip)
@@ -2949,13 +2949,11 @@ object AsOfJoin {
private def operandJoinSide(
expr: Expression,
leftSet: AttributeSet,
- rightSet: AttributeSet,
- syntacticIsLeft: Boolean): Option[Boolean] = {
+ rightSet: AttributeSet): Option[Boolean] = {
val refs = expr.references
if (refs.isEmpty) {
- // Literals, CURRENT_TIMESTAMP(), session variables, etc. have no column refs;
- // use MATCH_CONDITION syntactic position (expr1/expr2) for join-side assignment.
- Some(syntacticIsLeft)
+ // Constant operand (literal, current_timestamp(), session variable): no join input.
+ None
} else if (refs.subsetOf(leftSet)) {
Some(true)
} else if (refs.subsetOf(rightSet)) {
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index 6ad8b7a6acbe..46a950988d0d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -5163,8 +5163,8 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
new AnalysisException(
errorClass = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
messageParameters = Map(
- "refs1" -> toSQLExpr(expr1),
- "refs2" -> toSQLExpr(expr2)))
+ "operand1" -> toSQLExpr(expr1),
+ "operand2" -> toSQLExpr(expr2)))
}
def nestedSequentialStreamingUnionError(): Throwable = {
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-errors.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-errors.sql.out
index 65bcb1ab1d3e..7bcd343b4141 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-errors.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-errors.sql.out
@@ -371,8 +371,8 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"(trade_time + (quote_time - trade_time))\"",
- "refs2" : "\"quote_time\""
+ "operand1" : "\"(trade_time + (quote_time - trade_time))\"",
+ "operand2" : "\"quote_time\""
},
"queryContext" : [ {
"objectType" : "",
@@ -394,8 +394,8 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"trade_time\"",
- "refs2" : "\"trade_time - INTERVAL '01' HOUR\""
+ "operand1" : "\"trade_time\"",
+ "operand2" : "\"trade_time - INTERVAL '01' HOUR\""
}
}
@@ -534,7 +534,71 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"quote_time\"",
- "refs2" : "\"quote_time - INTERVAL '01' HOUR\""
+ "operand1" : "\"quote_time\"",
+ "operand2" : "\"quote_time - INTERVAL '01' HOUR\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"trade_time\"",
+ "operand2" : "\"TIMESTAMP '2026-06-29 10:00:00'\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
+ ON t.symbol = q.symbol
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"TIMESTAMP '2026-06-29 10:00:00'\"",
+ "operand2" : "\"quote_time\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (current_timestamp() >= q.quote_time)
+ ON t.symbol = q.symbol
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"current_timestamp()\"",
+ "operand2" : "\"quote_time\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:01' >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"TIMESTAMP '2026-06-29 10:00:01'\"",
+ "operand2" : "\"TIMESTAMP '2026-06-29 10:00:00'\""
}
}
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-expressions.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-expressions.sql.out
index 3fed752c4602..d08c8aacd123 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-expressions.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/join-asof-expressions.sql.out
@@ -195,33 +195,6 @@ org.apache.spark.sql.AnalysisException
}
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (current_timestamp() >= q.quote_time)
- ON t.symbol = q.symbol
--- !query analysis
-[Analyzer test output redacted due to nondeterminism]
-
-
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
- ON t.symbol = q.symbol
--- !query analysis
-[Analyzer test output redacted due to nondeterminism]
-
-
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
- ON t.symbol = q.symbol
--- !query analysis
-[Analyzer test output redacted due to nondeterminism]
-
-
-- !query
SELECT count(*) AS cnt
FROM trades t ASOF JOIN quotes q
diff --git a/sql/core/src/test/resources/sql-tests/inputs/join-asof-errors.sql b/sql/core/src/test/resources/sql-tests/inputs/join-asof-errors.sql
index df2d157c2e04..4b02ef7d8e32 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/join-asof-errors.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/join-asof-errors.sql
@@ -134,3 +134,23 @@ SELECT * FROM trades t ASOF JOIN quotes q
SELECT * FROM trades t ASOF JOIN quotes q
MATCH_CONDITION (q.quote_time >= q.quote_time - INTERVAL 1 HOUR)
ON t.symbol = q.symbol;
+
+-- FVT-ASOF-3-024: literal constant operand (right) references no join input
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol;
+
+-- FVT-ASOF-3-025: literal constant operand (left) references no join input
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
+ ON t.symbol = q.symbol;
+
+-- FVT-ASOF-3-026: query-foldable constant operand (current_timestamp) references no join input
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (current_timestamp() >= q.quote_time)
+ ON t.symbol = q.symbol;
+
+-- FVT-ASOF-3-027: both operands are constants that reference no join input
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:01' >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol;
diff --git a/sql/core/src/test/resources/sql-tests/inputs/join-asof-expressions.sql b/sql/core/src/test/resources/sql-tests/inputs/join-asof-expressions.sql
index 50b74c8b01f8..976db528dafc 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/join-asof-expressions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/join-asof-expressions.sql
@@ -68,23 +68,7 @@ SELECT * FROM trades t ASOF JOIN quotes q
MATCH_CONDITION (jitter_ts(t.trade_time) >= q.quote_time)
ON t.symbol = q.symbol;
--- FVT-ASOF-8-008: CURRENT_TIMESTAMP in MATCH_CONDITION operand (query-foldable constant)
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (current_timestamp() >= q.quote_time)
- ON t.symbol = q.symbol;
-
--- FVT-ASOF-8-008a: literal constant in MATCH_CONDITION operand (right)
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
- ON t.symbol = q.symbol;
-
--- FVT-ASOF-8-008b: literal constant in MATCH_CONDITION operand (left)
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
- ON t.symbol = q.symbol;
+-- FVT-ASOF-8-008: constant MATCH_CONDITION operand, covered by FVT-ASOF-3-024..027 in join-asof-errors.sql
-- FVT-ASOF-8-009: QUALIFY on ASOF — covered by FVT-ASOF-6-007
diff --git a/sql/core/src/test/resources/sql-tests/results/join-asof-errors.sql.out b/sql/core/src/test/resources/sql-tests/results/join-asof-errors.sql.out
index cb3e2cb53dbc..3233f503da8d 100644
--- a/sql/core/src/test/resources/sql-tests/results/join-asof-errors.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/join-asof-errors.sql.out
@@ -399,8 +399,8 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"(trade_time + (quote_time - trade_time))\"",
- "refs2" : "\"quote_time\""
+ "operand1" : "\"(trade_time + (quote_time - trade_time))\"",
+ "operand2" : "\"quote_time\""
},
"queryContext" : [ {
"objectType" : "",
@@ -424,8 +424,8 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"trade_time\"",
- "refs2" : "\"trade_time - INTERVAL '01' HOUR\""
+ "operand1" : "\"trade_time\"",
+ "operand2" : "\"trade_time - INTERVAL '01' HOUR\""
}
}
@@ -578,7 +578,79 @@ org.apache.spark.sql.AnalysisException
"errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
"sqlState" : "42K0E",
"messageParameters" : {
- "refs1" : "\"quote_time\"",
- "refs2" : "\"quote_time - INTERVAL '01' HOUR\""
+ "operand1" : "\"quote_time\"",
+ "operand2" : "\"quote_time - INTERVAL '01' HOUR\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"trade_time\"",
+ "operand2" : "\"TIMESTAMP '2026-06-29 10:00:00'\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
+ ON t.symbol = q.symbol
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"TIMESTAMP '2026-06-29 10:00:00'\"",
+ "operand2" : "\"quote_time\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (current_timestamp() >= q.quote_time)
+ ON t.symbol = q.symbol
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"current_timestamp()\"",
+ "operand2" : "\"quote_time\""
+ }
+}
+
+
+-- !query
+SELECT * FROM trades t ASOF JOIN quotes q
+ MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:01' >= TIMESTAMP '2026-06-29 10:00:00')
+ ON t.symbol = q.symbol
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" : "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ "sqlState" : "42K0E",
+ "messageParameters" : {
+ "operand1" : "\"TIMESTAMP '2026-06-29 10:00:01'\"",
+ "operand2" : "\"TIMESTAMP '2026-06-29 10:00:00'\""
}
}
diff --git a/sql/core/src/test/resources/sql-tests/results/join-asof-expressions.sql.out b/sql/core/src/test/resources/sql-tests/results/join-asof-expressions.sql.out
index fd9b7740daff..536eb98f705a 100644
--- a/sql/core/src/test/resources/sql-tests/results/join-asof-expressions.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/join-asof-expressions.sql.out
@@ -157,39 +157,6 @@ org.apache.spark.sql.AnalysisException
}
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (current_timestamp() >= q.quote_time)
- ON t.symbol = q.symbol
--- !query schema
-struct<cnt:bigint>
--- !query output
-3
-
-
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
- ON t.symbol = q.symbol
--- !query schema
-struct<cnt:bigint>
--- !query output
-3
-
-
--- !query
-SELECT count(*) AS cnt
-FROM trades t ASOF JOIN quotes q
- MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= q.quote_time)
- ON t.symbol = q.symbol
--- !query schema
-struct<cnt:bigint>
--- !query output
-2
-
-
-- !query
SELECT count(*) AS cnt
FROM trades t ASOF JOIN quotes q
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSQLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSQLSuite.scala
index 776cfd00f2ff..8efe157ca3f6 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSQLSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSQLSuite.scala
@@ -134,8 +134,8 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
sqlState = Some("42K0E"),
parameters = Map(
- "refs1" -> "\"(trade_time + quote_time)\"",
- "refs2" -> "\"quote_time\""),
+ "operand1" -> "\"(trade_time + quote_time)\"",
+ "operand2" -> "\"quote_time\""),
queryContext = Array(
ExpectedContext(
fragment = """ASOF JOIN quotes q
@@ -159,14 +159,13 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
sqlState = Some("42K0E"),
parameters = Map(
- "refs1" -> "\"symbol\"",
- "refs2" -> "\"symbol\""))
+ "operand1" -> "\"symbol\"",
+ "operand2" -> "\"symbol\""))
}
- test("MATCH_CONDITION rejects a literal colliding with a same-side column") {
+ test("MATCH_CONDITION rejects a literal even when a same-side column is present") {
setupTradeQuoteViews()
- // `'AAPL'` has no column refs, so it takes its syntactic (right) side, the same side as
- // q.symbol. No operand references the left input, so the condition is rejected.
+ // `'AAPL'` references no join input, so no operand binds to the left side and it is rejected.
val sqlText =
"""
|SELECT count(*)
@@ -178,7 +177,7 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
exception = intercept[AnalysisException](sql(sqlText)),
condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
sqlState = Some("42K0E"),
- parameters = Map("refs1" -> "\"symbol\"", "refs2" -> "\"AAPL\""))
+ parameters = Map("operand1" -> "\"symbol\"", "operand2" -> "\"AAPL\""))
}
test("MATCH_CONDITION rejects non-deterministic expressions") {
@@ -277,7 +276,7 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
}
}
- test("MATCH_CONDITION accepts CURRENT_TIMESTAMP as left operand") {
+ test("MATCH_CONDITION rejects a query-foldable constant operand (no join input reference)") {
setupTradeQuoteViews()
val sqlText =
"""
@@ -286,15 +285,16 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
| MATCH_CONDITION (current_timestamp() >= q.quote_time)
| ON t.symbol = q.symbol
|""".stripMargin
- val asOfJoin = sql(sqlText).queryExecution.analyzed.collectFirst {
- case j: AsOfJoin => j
- }.get
- assert(asOfJoin.asOfCondition.resolved)
- assert(asOfJoin.leftSortExprs.nonEmpty)
- assert(asOfJoin.rightSortExprs.nonEmpty)
+ checkError(
+ exception = intercept[AnalysisException](sql(sqlText)),
+ condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ sqlState = Some("42K0E"),
+ parameters = Map(
+ "operand1" -> "\"current_timestamp()\"",
+ "operand2" -> "\"quote_time\""))
}
- test("MATCH_CONDITION accepts literal constant as right operand") {
+ test("MATCH_CONDITION rejects a literal constant operand (no join input reference)") {
setupTradeQuoteViews()
val sqlText =
"""
@@ -303,14 +303,18 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
| MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
| ON t.symbol = q.symbol
|""".stripMargin
- val asOfJoin = sql(sqlText).queryExecution.analyzed.collectFirst {
- case j: AsOfJoin => j
- }.get
- assert(asOfJoin.asOfCondition.resolved)
+ checkError(
+ exception = intercept[AnalysisException](sql(sqlText)),
+ condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ sqlState = Some("42K0E"),
+ parameters = Map(
+ "operand1" -> "\"trade_time\"",
+ "operand2" -> "\"TIMESTAMP '2026-06-29 10:00:00'\""))
}
- test("MATCH_CONDITION with two literal operands passes analysis") {
+ test("MATCH_CONDITION rejects two literal operands (neither references a join input)") {
setupTradeQuoteViews()
+ // Both operands are literals that reference no join input, so neither binds to a side.
val sqlText =
"""
|SELECT count(*)
@@ -318,13 +322,33 @@ class AsOfJoinSQLSuite extends QueryTest with SharedSparkSession {
| MATCH_CONDITION (TIMESTAMP '2026-06-29 10:00:00' >= TIMESTAMP '2026-06-29 09:00:00')
| ON t.symbol = q.symbol
|""".stripMargin
- // Both operands are literals with no column refs, so each takes its syntactic side
- // (expr1 = left, expr2 = right). Opposite sides pass the table-reference check; same-side
- // assignment would throw ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE before this point.
- val asOfJoin = sql(sqlText).queryExecution.analyzed.collectFirst {
- case j: AsOfJoin => j
- }.get
- assert(asOfJoin.asOfCondition.resolved)
+ checkError(
+ exception = intercept[AnalysisException](sql(sqlText)),
+ condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ sqlState = Some("42K0E"),
+ parameters = Map(
+ "operand1" -> "\"TIMESTAMP '2026-06-29 10:00:00'\"",
+ "operand2" -> "\"TIMESTAMP '2026-06-29 09:00:00'\""))
+ }
+
+ test("MATCH_CONDITION rejects a constant operand under the single-pass analyzer") {
+ setupTradeQuoteViews()
+ val sqlText =
+ """
+ |SELECT count(*)
+ |FROM trades t ASOF JOIN quotes q
+ | MATCH_CONDITION (t.trade_time >= TIMESTAMP '2026-06-29 10:00:00')
+ | ON t.symbol = q.symbol
+ |""".stripMargin
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ checkError(
+ exception = intercept[AnalysisException](sql(sqlText)),
+ condition = "ASOF_JOIN_MATCH_CONDITION_TABLE_REFERENCE",
+ sqlState = Some("42K0E"),
+ parameters = Map(
+ "operand1" -> "\"trade_time\"",
+ "operand2" -> "\"TIMESTAMP '2026-06-29 10:00:00'\""))
+ }
}
test("MATCH_CONDITION rejects scalar subquery operand") {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org