Re: [I] Check for read errors when hoodie.schema.on.read is enabled in a table with Variants [hudi]

["voonhous (via GitHub)" <gi...@apache.org>]

voonhous commented on issue #18021:
URL: https://github.com/apache/hudi/issues/18021#issuecomment-5354806993

   Verified empirically on Spark 3.5 (scala-2.12), reading a Spark 4.1-written COW variant table that carries a committed internal schema (schema-on-read DDL). Three legs:
   
   | Read | Outcome |
   |---|---|
   | Auto-resolve, plain | `HoodieSchemaException` caused by `VARIANT type is only supported in Spark 4.0+` (the known, pinned rejection) |
   | Auto-resolve, `hoodie.schema.on.read.enable=true` | Same exception, same cause - the InternalSchema path (sentinel round trip restores the VARIANT logical type) fails identically, not worse |
   | Struct-DDL compat mode + `hoodie.schema.on.read.enable=true` | Same exception - internal-schema resolution overrides the user DDL, so the documented Spark 3.x compat mode breaks when the conf is on |
   
   So the answer to this task: the path throws, but always loudly and with the actionable message - no silent wrong data and no obscure secondary failure under `toSqlTypeHelper`. The one real caveat is the third row: Spark 3.x compat-mode readers must keep `hoodie.schema.on.read.enable` off for variant tables that have a committed internal schema; real support belongs to #18285.
   
   Pinned by a new Spark 3-gated test (`TestVariantDataType`, runs in the spark3.5 CI lane) over a new fixture `variant_backward_compat/variant_schema_on_read_cow.zip`, on branch `verify-18021-sor-variant-spark35`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org