You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/05/25 14:26:00 UTC
[PR] docs(variant): add Spark 3.5 backward-compat read guide [hudi]
voonhous opened a new pull request, #18839:
URL: https://github.com/apache/hudi/pull/18839
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
Closes #18333.
The existing variant_type.md does not explain how to read a Hudi 1.2.0 VARIANT table from Spark 3.5 — users hit HoodieSchemaException ("VARIANT type is only supported in Spark 4.0+") when relying on schema auto-resolution.
Add a "Reading from Spark 3.5 (Backward Compatibility)" section documenting the supported path: pre-declare the table with
STRUCT<value: BINARY, metadata: BINARY> against the existing table location, read raw bytes, and decode in application code (or upgrade to Spark 4.0+ for native VARIANT). Also cross-reference the new section from the Spark 3.x DDL tab and the Cross-Engine Compatibility table.
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(variant): add Spark 3.5 backward-compat read guide [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #18839:
URL: https://github.com/apache/hudi/pull/18839#discussion_r3302534333
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine compatible |
A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and vice versa. The
binary encoding is engine-independent.
+## Reading from Spark 3.5 (Backward Compatibility)
+
+Spark 3.5 cannot construct a native `VariantType`, so it cannot consume a VARIANT-typed Hudi
+table the same way Spark 4.0+ does. This section covers the supported read path: how to point
+a Spark 3.5 job at a 1.2.0 VARIANT table, what you get back, and what you give up.
+
+### Why a special path is needed
+
+If you let Hudi resolve the table schema from commit metadata on Spark 3.5, the read fails
+fast because the Spark 3.x adapter rejects the `VariantType` conversion:
+
+```python
+spark.read.format("hudi").load("/path/to/events").show()
+# org.apache.hudi.exception.HoodieSchemaException: ...
+# Caused by: java.lang.UnsupportedOperationException:
+# VARIANT type is only supported in Spark 4.0+
+```
+
+The table data on disk is fine — Hudi just needs to be told the column's *physical* shape
+on the reader side so it can skip the unsupported logical conversion.
+
+### Pre-declare the table with a Variant-shaped struct
+
+Create a Hudi table at the existing table path, declaring the VARIANT column as a struct of
+two binary fields:
+
+```sql
+CREATE TABLE events (
+ event_id STRING,
+ payload STRUCT<value: BINARY, metadata: BINARY>,
+ ts BIGINT
+) USING hudi
+LOCATION '/path/to/events'
+TBLPROPERTIES (
+ primaryKey = 'event_id',
+ preCombineField = 'ts',
+ type = 'cow' -- or 'mor'
+);
+
+SELECT event_id, payload.value, payload.metadata FROM events;
+```
+
+Hudi recognizes `STRUCT<value: BINARY, metadata: BINARY>` as a logical VARIANT (via the
+`hudi_type=VARIANT` field-metadata tag stamped at write time) and routes the read through
+the binary path. This works against both COW tables and MOR tables — both `AVRO` and `SPARK`
+log-record types — because compaction and log-merge operate on the binary representation.
+
+### What you get, and what you don't
+
+On Spark 3.5 you get the raw VARIANT binary buffers:
+
+- `payload.metadata` — the field-name dictionary (binary).
+- `payload.value` — the encoded payload (binary).
+
+Both conform to the open [Apache Spark Variant binary
Review Comment:
Our current code does not support shredding, so we should also state that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(variant): add Spark 3.5 backward-compat read guide [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #18839:
URL: https://github.com/apache/hudi/pull/18839#discussion_r3302511048
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
Review Comment:
Addressed
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine compatible |
A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and vice versa. The
binary encoding is engine-independent.
+## Reading from Spark 3.5 (Backward Compatibility)
+
+Spark 3.5 cannot construct a native `VariantType`, so it cannot consume a VARIANT-typed Hudi
+table the same way Spark 4.0+ does. This section covers the supported read path: how to point
+a Spark 3.5 job at a 1.2.0 VARIANT table, what you get back, and what you give up.
+
+### Why a special path is needed
+
+If you let Hudi resolve the table schema from commit metadata on Spark 3.5, the read fails
+fast because the Spark 3.x adapter rejects the `VariantType` conversion:
+
+```python
+spark.read.format("hudi").load("/path/to/events").show()
+# org.apache.hudi.exception.HoodieSchemaException: ...
+# Caused by: java.lang.UnsupportedOperationException:
+# VARIANT type is only supported in Spark 4.0+
+```
+
+The table data on disk is fine — Hudi just needs to be told the column's *physical* shape
+on the reader side so it can skip the unsupported logical conversion.
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(variant): add Spark 3.5 backward-compat read guide [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #18839:
URL: https://github.com/apache/hudi/pull/18839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(variant): add Spark 3.5 backward-compat read guide [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #18839:
URL: https://github.com/apache/hudi/pull/18839#discussion_r3298691005
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
Review Comment:
🤖 The Cross-Engine Compatibility table still labels this row **Spark 3.x**, but the new section it links to is titled **Reading from Spark 3.5**. It might help to clarify whether the workaround applies to all Spark 3.x readers Hudi supports (3.3, 3.4, 3.5) or only to 3.5 — readers on 3.3/3.4 will likely follow this link expecting their version to be covered. If the path is identical for all 3.x, consider renaming the section heading to "Reading from Spark 3.x"; if 3.5 is the only tested target, calling that out explicitly in the section intro would prevent confusion.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine compatible |
A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and vice versa. The
binary encoding is engine-independent.
+## Reading from Spark 3.5 (Backward Compatibility)
+
+Spark 3.5 cannot construct a native `VariantType`, so it cannot consume a VARIANT-typed Hudi
+table the same way Spark 4.0+ does. This section covers the supported read path: how to point
+a Spark 3.5 job at a 1.2.0 VARIANT table, what you get back, and what you give up.
+
+### Why a special path is needed
+
+If you let Hudi resolve the table schema from commit metadata on Spark 3.5, the read fails
+fast because the Spark 3.x adapter rejects the `VariantType` conversion:
+
+```python
+spark.read.format("hudi").load("/path/to/events").show()
+# org.apache.hudi.exception.HoodieSchemaException: ...
+# Caused by: java.lang.UnsupportedOperationException:
+# VARIANT type is only supported in Spark 4.0+
+```
+
+The table data on disk is fine — Hudi just needs to be told the column's *physical* shape
+on the reader side so it can skip the unsupported logical conversion.
+
+### Pre-declare the table with a Variant-shaped struct
+
+Create a Hudi table at the existing table path, declaring the VARIANT column as a struct of
+two binary fields:
+
+```sql
+CREATE TABLE events (
+ event_id STRING,
+ payload STRUCT<value: BINARY, metadata: BINARY>,
+ ts BIGINT
+) USING hudi
+LOCATION '/path/to/events'
+TBLPROPERTIES (
+ primaryKey = 'event_id',
+ preCombineField = 'ts',
+ type = 'cow' -- or 'mor'
+);
+
+SELECT event_id, payload.value, payload.metadata FROM events;
+```
+
+Hudi recognizes `STRUCT<value: BINARY, metadata: BINARY>` as a logical VARIANT (via the
+`hudi_type=VARIANT` field-metadata tag stamped at write time) and routes the read through
+the binary path. This works against both COW tables and MOR tables — both `AVRO` and `SPARK`
+log-record types — because compaction and log-merge operate on the binary representation.
+
+### What you get, and what you don't
+
+On Spark 3.5 you get the raw VARIANT binary buffers:
+
+- `payload.metadata` — the field-name dictionary (binary).
+- `payload.value` — the encoded payload (binary).
+
+Both conform to the open [Apache Spark Variant binary
Review Comment:
🤖 The caveat about shredded VARIANT tables says users "must declare [`typed_value` sub-columns] explicitly in your DDL" but the example DDL above only shows the unshredded `STRUCT<value: BINARY, metadata: BINARY>` shape. A one-line example of the shredded struct shape (or a forward reference to the shredding doc) would make this actionable — otherwise a reader on a shredded table won't know what shape to write. @yihua could you confirm whether the shredded struct layout is stable enough to document here for the 1.2.0 release?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
website/docs/variant_type.md:
##########
@@ -254,12 +257,94 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
| **Spark 4.0+** | Native `VariantType` — full read/write/query |
-| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — backward compatible |
+| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` — see [Reading from Spark 3.5](#reading-from-spark-35-backward-compatibility) |
| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine compatible |
A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and vice versa. The
binary encoding is engine-independent.
+## Reading from Spark 3.5 (Backward Compatibility)
+
+Spark 3.5 cannot construct a native `VariantType`, so it cannot consume a VARIANT-typed Hudi
+table the same way Spark 4.0+ does. This section covers the supported read path: how to point
+a Spark 3.5 job at a 1.2.0 VARIANT table, what you get back, and what you give up.
+
+### Why a special path is needed
+
+If you let Hudi resolve the table schema from commit metadata on Spark 3.5, the read fails
+fast because the Spark 3.x adapter rejects the `VariantType` conversion:
+
+```python
+spark.read.format("hudi").load("/path/to/events").show()
+# org.apache.hudi.exception.HoodieSchemaException: ...
+# Caused by: java.lang.UnsupportedOperationException:
+# VARIANT type is only supported in Spark 4.0+
+```
+
+The table data on disk is fine — Hudi just needs to be told the column's *physical* shape
+on the reader side so it can skip the unsupported logical conversion.
Review Comment:
🤖 When pre-declaring a table at an existing Hudi LOCATION, what is the expected behavior if `primaryKey`, `preCombineField`, or `type` in `TBLPROPERTIES` differ from what was stamped into the existing `hoodie.properties` at write time? It would be useful to either (a) state that these properties must match the existing table and point readers to `.hoodie/hoodie.properties` to find the right values, or (b) clarify that Hudi will ignore/override them from the on-disk metadata. As written, a user copy-pasting this DDL against a table whose preCombineField isn't `ts` (or whose type is `mor`) may either silently get wrong behavior or hit a confusing error.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org