[I] fix(spark): extended SQL parser enforces ANSI reserved keywords on Spark 4.x, rejecting TIMESTAMP columns in BLOB/VECTOR DDL [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous opened a new issue, #19449:
URL: https://github.com/apache/hudi/issues/19449

   **Describe the problem you faced**
   
   All six per-version extended SQL parsers set
   
   ```scala
   parser.SQL_standard_keyword_behavior = conf.ansiEnabled
   ```
   
   (`HoodieSpark3_3ExtendedSqlParser.scala:88` through `HoodieSpark4_2ExtendedSqlParser.scala:90`), but Spark's own parser wires this flag to `conf.enforceReservedKeywords` (`spark.sql.ansi.enforceReservedKeywords`), which defaults to **false** even when ANSI mode is on.
   
   Spark 4.x defaults `spark.sql.ansi.enabled=true`, so the forked parser enforces reserved keywords that stock Spark leaves off. Any statement routed to the extended parser (anything containing a ` blob` or ` vector` token per `isHoodieCommand`) then rejects SQL the stock parser accepts:
   
   ```sql
   CREATE TABLE t (id BIGINT, ts TIMESTAMP, data BLOB) USING hudi
   -- Spark 4.x: ParseException: no viable alternative at input 'TIMESTAMP'
   ```
   
   The same statement without the BLOB column parses fine. Blast radius includes a bare `TIMESTAMP` column type, `TIMESTAMP '...'` literals, and ANSI-reserved column names (`user`, `left`, `filter`, `some`, `current_date`).
   
   **Fix**
   
   One line in each of the six parsers: set the flag from `conf.enforceReservedKeywords` instead of `conf.ansiEnabled`, matching Spark's `AbstractParser`. The conf exists since Spark 3.3.0, so it is safe on all supported profiles.
   
   The 3.5 and regenerated 4.1 forked grammars behave identically once the flag matches -- the grammar itself has no gap.
   
   Found while reviewing #19408; its tests use `DATE` instead of `TIMESTAMP` to work around this.
   
   **Environment Description**
   
   * Hudi version: master
   * Spark version: 4.0/4.1/4.2 profiles (any profile with `spark.sql.ansi.enabled=true`)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] fix(spark): extended SQL parser enforces ANSI reserved keywords on Spark 4.x, rejecting TIMESTAMP columns in BLOB/VECTOR DDL [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous closed issue #19449: fix(spark): extended SQL parser enforces ANSI reserved keywords on Spark 4.x, rejecting TIMESTAMP columns in BLOB/VECTOR DDL
URL: https://github.com/apache/hudi/issues/19449


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org