You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "anew (via GitHub)" <gi...@apache.org> on 2026/07/22 18:32:52 UTC
[PR] [WIP][SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
anew opened a new pull request, #57434:
URL: https://github.com/apache/spark/pull/57434
### What changes were proposed in this pull request?
Add SQL syntax for selecting the SCD type and history-tracking columns of an AUTO CDC flow, mirroring the recently added Python/Connect surface. Both SQL AUTO CDC forms -- `CREATE STREAMING TABLE ... FLOW AUTO CDC ...` and `CREATE FLOW ... AS AUTO CDC INTO ...` -- now accept:
- `STORED AS SCD TYPE 1 | 2` (defaults to SCD Type 1 when omitted), and
- `TRACK HISTORY ON (<cols>)` / `TRACK HISTORY ON * EXCEPT (<cols>)` under SCD Type 2.
Specifically:
- Grammar: add multi-word lexer tokens `SCD_TYPE_1`/`SCD_TYPE_2` and `HISTORY`/`TRACK` keywords; add `autoCdcStoredAsClause` and `autoCdcTrackHistoryClause`, appended in fixed order after the existing `COLUMNS` clause. `HISTORY`/`TRACK` are registered as non-reserved.
- AST/plans: thread `storedAsScdType` (Int, default 1) and the track-history column lists through `AutoCdcParams`, `AutoCdcInto`, and `CreateStreamingTableAutoCdc`.
- Registration: `SqlGraphRegistrationContext.buildChangeArgs` maps the SCD type onto `ScdType`, builds `trackHistorySelection`, and rejects TRACK HISTORY under SCD1 as well as specifying both track-history lists.
The SCD type and track-history columns feed the existing `ChangeArgs` model (`storedAsScdType` / `trackHistorySelection`), the same contract the engine's SCD2 execution work consumes.
### Why are the changes needed?
SQL AUTO CDC previously only supported SCD Type 1 and rejected `STORED AS SCD TYPE 2` / `TRACK HISTORY` at parse time. SCD Type 2 (history tracking) is a common CDC requirement and is already modeled in the engine's `ChangeArgs` / `ScdType`, so the SQL surface was the missing piece.
### Does this PR introduce any user-facing change?
Yes. SQL AUTO CDC statements accept `STORED AS SCD TYPE 1|2` and, under SCD Type 2, `TRACK HISTORY ON ...`. Previously these clauses failed to parse. This is a change within the unreleased master branch only.
### How was this patch tested?
New parser tests in `AutoCdcParserSuite` (SCD type 1/2, TRACK HISTORY include/except, all clauses combined, clause-ordering and invalid-type negative cases) and registration tests in `SqlPipelineSuite` (SCD2 and track-history map onto `ChangeArgs`; TRACK HISTORY without SCD2 is rejected). Ran `AutoCdcParserSuite`, `SQLKeywordSuite`, and the AUTO CDC `SqlPipelineSuite` cases.
### Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #57434:
URL: https://github.com/apache/spark/pull/57434#issuecomment-5052812913
Two test additions would be good (either in `AutoCdcParserSuite` or the pipeline suite):
1. Explicit `STORED AS SCD TYPE 1` combined with `TRACK HISTORY` is rejected -- currently only the implicit default-SCD1 + `TRACK HISTORY` case is covered.
2. Identifiers named after the new keywords still parse -- e.g. columns/tables named `history` and `track` (and `scd`, if the single-word `SCD` keyword suggested on the lexer is adopted) -- to guard the non-reserved classification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634742268
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -2294,6 +2307,8 @@ ansiNonReserved
| ROWS
| SCHEMA
| SCHEMAS
+ | SCD_TYPE_1
Review Comment:
These two (and the matching `SCD_TYPE_1`/`SCD_TYPE_2` pair added to `nonReserved` below) can just be dropped from the non-reserved lists -- they don't serve the usual purpose, and removing them is safe:
- Non-reserved only governs whether a token can be used as a bare identifier, but `SCD_TYPE_1`/`SCD_TYPE_2` are multi-word literals (`'SCD TYPE 1'`), so they can never stand in as an identifier anyway.
- `autoCdcStoredAsClause` references the tokens directly, so parsing `STORED AS SCD TYPE 1|2` doesn't depend on their non-reserved membership.
- `SQLKeywordSuite`'s extractor regex is `[A-Z_]+`, which the digit in the token name fails, so these are already excluded from `allCandidateKeywords` and the computed non-reserved sets -- listing them here has no effect on the suite, and they were (correctly) never added to the docs table or the `keywords*.sql.out` golden files.
- They're new in this PR, so nothing external depends on them.
Net: it's a pure deletion with no docs/golden/test changes, and `scd` stays a valid identifier either way (it isn't a keyword in this approach).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #57434:
URL: https://github.com/apache/spark/pull/57434#issuecomment-5061834168
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/c052ccaf8ccd431f02df80c0c48e7018f691010d
- merged into branch-4.x https://github.com/apache/spark/commit/d22c7e43593c0f88cc7fa97cc74f6866ac47c114
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "anew (via GitHub)" <gi...@apache.org>.
anew commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634086684
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4:
##########
@@ -460,6 +461,8 @@ SECOND: 'SECOND';
SECONDS: 'SECONDS';
SCHEMA: 'SCHEMA';
SCHEMAS: 'SCHEMAS';
+SCD_TYPE_1: 'SCD TYPE 1';
Review Comment:
This is suboptimal as it expects excatly one space between the words.
I filed https://issues.apache.org/jira/browse/SPARK-58270 for future improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634698765
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4:
##########
@@ -460,6 +461,8 @@ SECOND: 'SECOND';
SECONDS: 'SECONDS';
SCHEMA: 'SCHEMA';
SCHEMAS: 'SCHEMAS';
+SCD_TYPE_1: 'SCD TYPE 1';
Review Comment:
Follow-up to SPARK-58270: a single-word `SCD` keyword would sidestep the one-space limitation entirely -- the lexer would tokenize `SCD`, `TYPE`, and the number independently, so any whitespace between them works -- reusing the existing `TYPE` and `INTEGER_VALUE` tokens for the rest of the clause. `SCD` would then just need to be added to the `nonReserved` / `ansiNonReserved` lists so `scd` stays usable as a bare identifier (same reason `HISTORY`/`TRACK` were added). Note the current multi-word `SCD_TYPE_1`/`SCD_TYPE_2` entries in those lists can never be identifiers, so they don't serve that purpose today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "anew (via GitHub)" <gi...@apache.org>.
anew commented on PR #57434:
URL: https://github.com/apache/spark/pull/57434#issuecomment-5053112087
Added the two new tests: SqlPipelineSuite now covers explicit STORED AS SCD TYPE 1 + TRACK HISTORY rejection (alongside the implicit case), and AutoCdcParserSuite has a test using history, track, and scd as table/column identifiers to guard their non-reserved status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634685546
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4:
##########
@@ -460,6 +461,8 @@ SECOND: 'SECOND';
SECONDS: 'SECONDS';
SCHEMA: 'SCHEMA';
SCHEMAS: 'SCHEMAS';
+SCD_TYPE_1: 'SCD TYPE 1';
Review Comment:
This could be avoided (and SPARK-58270 resolved here) by composing the clause from single-word tokens instead of the multi-word `SCD_TYPE_1`/`SCD_TYPE_2` literals. `TYPE` and `INTEGER_VALUE` already exist, so only a new `SCD` keyword is needed:
```antlr
SCD: 'SCD';
autoCdcStoredAsClause
: STORED AS SCD TYPE scdType=INTEGER_VALUE
;
```
then validate `scdType in {1, 2}` in `parseAutoCdcParams`. Benefits:
- The lexer tokenizes `SCD` / `TYPE` / `1` independently, so arbitrary whitespace (and comments) between the words just works, without the exactly-one-space limitation.
- `SCD` becomes a normal single-word keyword that flows through the usual non-reserved machinery (docs + keyword golden files), instead of `SCD_TYPE_1`/`SCD_TYPE_2` sitting in `ansiNonReserved`/`nonReserved`, where they can never be identifiers and are silently skipped by `SQLKeywordSuite` anyway. Only `SCD` needs adding to the non-reserved lists (to keep `scd` usable as a bare identifier, same reason `HISTORY`/`TRACK` were added).
- `STORED AS SCD TYPE 3` then fails with a clear AST-level validation message rather than a generic `PARSE_SYNTAX_ERROR`, matching the defensive `case other` already in `buildChangeArgs`.
A couple of test additions would also help:
- Explicit `STORED AS SCD TYPE 1` combined with `TRACK HISTORY` is rejected -- currently only the implicit default-SCD1 + `TRACK HISTORY` case is covered.
- `scd` (and `history` / `track`) still parse as identifiers, e.g. a column or table named `scd`, to guard the non-reserved classification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634698765
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4:
##########
@@ -460,6 +461,8 @@ SECOND: 'SECOND';
SECONDS: 'SECONDS';
SCHEMA: 'SCHEMA';
SCHEMAS: 'SCHEMAS';
+SCD_TYPE_1: 'SCD TYPE 1';
Review Comment:
Follow-up to SPARK-58270: a single-word `SCD` keyword would sidestep the one-space limitation entirely -- the lexer would tokenize `SCD`, `TYPE`, and the number independently, so any whitespace between them works -- reusing the existing `TYPE` and `INTEGER_VALUE` tokens for the rest of the clause. `SCD` would then just need to be added to the `nonReserved` / `ansiNonReserved` lists so `scd` stays usable as a bare identifier (same reason `HISTORY`/`TRACK` were added). Note the current multi-word `SCD_TYPE_1`/`SCD_TYPE_2` entries in those lists can never be identifiers, so they don't serve that purpose today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "anew (via GitHub)" <gi...@apache.org>.
anew commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634847539
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -2294,6 +2307,8 @@ ansiNonReserved
| ROWS
| SCHEMA
| SCHEMAS
+ | SCD_TYPE_1
Review Comment:
Good call — went a step further and reworked the lexing: SCD TYPE n is now three separate tokens (SCD, TYPE, INTEGER_VALUE) validated in the AstBuilder, instead of multi-word literals. That drops the odd non-reserved entries you flagged, and as a bonus fixes the whitespace brittleness from SPARK-58270 (which can probably be closed). SCD TYPE 3 now gives a clear "Unsupported SCD type" error too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed pull request #57434: [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC
URL: https://github.com/apache/spark/pull/57434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634685546
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4:
##########
@@ -460,6 +461,8 @@ SECOND: 'SECOND';
SECONDS: 'SECONDS';
SCHEMA: 'SCHEMA';
SCHEMAS: 'SCHEMAS';
+SCD_TYPE_1: 'SCD TYPE 1';
Review Comment:
This could be avoided (and SPARK-58270 resolved here) by composing the clause from single-word tokens instead of the multi-word `SCD_TYPE_1`/`SCD_TYPE_2` literals. `TYPE` and `INTEGER_VALUE` already exist, so only a new `SCD` keyword is needed:
```antlr
SCD: 'SCD';
autoCdcStoredAsClause
: STORED AS SCD TYPE scdType=INTEGER_VALUE
;
```
then validate `scdType in {1, 2}` in `parseAutoCdcParams`. Benefits:
- The lexer tokenizes `SCD` / `TYPE` / `1` independently, so arbitrary whitespace (and comments) between the words just works, without the exactly-one-space limitation.
- `SCD` becomes a normal single-word keyword that flows through the usual non-reserved machinery (docs + keyword golden files), instead of `SCD_TYPE_1`/`SCD_TYPE_2` sitting in `ansiNonReserved`/`nonReserved`, where they can never be identifiers and are silently skipped by `SQLKeywordSuite` anyway. Only `SCD` needs adding to the non-reserved lists (to keep `scd` usable as a bare identifier, same reason `HISTORY`/`TRACK` were added).
- `STORED AS SCD TYPE 3` then fails with a clear AST-level validation message rather than a generic `PARSE_SYNTAX_ERROR`, matching the defensive `case other` already in `buildChangeArgs`.
A couple of test additions would also help:
- Explicit `STORED AS SCD TYPE 1` combined with `TRACK HISTORY` is rejected -- currently only the implicit default-SCD1 + `TRACK HISTORY` case is covered.
- `scd` (and `history` / `track`) still parse as identifiers, e.g. a column or table named `scd`, to guard the non-reserved classification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58261][SQL][SDP] Support SCD Type 2 syntax in SQL AUTO CDC [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #57434:
URL: https://github.com/apache/spark/pull/57434#discussion_r3634883574
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1430,13 +1433,40 @@ class AstBuilder extends DataTypeAstBuilder
visitIdentifierSeq(c.exceptCols).map(UnresolvedAttribute.quoted)
}
+ // STORED AS SCD TYPE <n>. Absent clause defaults to SCD Type 1. Only 1 and 2 are
+ // supported; reject anything else with a clear error rather than a generic parse failure.
+ val storedAsScdType = Option(params.autoCdcStoredAsClause()) match {
+ case Some(c) =>
+ val scdType = c.`type`.getText.toInt
Review Comment:
Minor robustness nit: `INTEGER_VALUE` is `DIGIT+`, so an overflowing literal such as `STORED AS SCD TYPE 99999999999999999999` makes `.toInt` throw an uncaught `NumberFormatException` -- which surfaces as an internal error/stack trace rather than the intended `Unsupported SCD type` message this block is trying to produce.
AstBuilder already guards this pattern elsewhere (the nearest-by `num`, ~L2627):
```scala
val value = try n.getText.toLong catch {
case _: NumberFormatException => throw ...outOfRange...
}
```
Since only `1` and `2` are valid, the simplest fix is to branch on the literal text instead of parsing to Int -- e.g. match `getText` on `"1"` / `"2"` and route everything else (including oversized numbers) through the existing `operationNotAllowed(...)`. That closes the overflow hole and keeps the clear error message.
##########
sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4:
##########
@@ -790,6 +792,16 @@ autoCdcColumnsClause
ASTERISK EXCEPT LEFT_PAREN exceptCols=identifierSeq RIGHT_PAREN)
;
+autoCdcStoredAsClause
+ : STORED AS SCD TYPE type=INTEGER_VALUE
Review Comment:
Minor: the label name `type` collides with the Scala keyword, forcing a backtick escape when accessing it in the AstBuilder. Renaming the label to `scdType=INTEGER_VALUE` reads better and avoids the escape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org