You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:52:59 UTC
[I] Data issue with hive sync if adding the column in between [hudi]
hudi-bot opened a new issue, #17364:
URL: https://github.com/apache/hudi/issues/17364
Reproducible Code
```
tableName = "trips_table"
basePath = "s3://<bucket>/results/temporary/trips_table_wmt"
columns = ["ts","uuid","rider","driver","fare","city"]
data =[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san_francisco"),
(1695091554788,"e96c4396-3fad-413a-a942-4cb36106d721","rider-C","driver-M",27.70 ,"san_francisco"),
(1695046462179,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-D","driver-L",33.90 ,"san_francisco"),
(1695516137016,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-F","driver-P",34.15,"sao_paulo"),
(1695115999911,"c8abbe79-8d89-47ea-b4ce-4d224bae5bfa","rider-J","driver-T",17.85,"chennai")]
inserts = spark.createDataFrame(data).toDF(*columns)
hudi_options =
{ 'hoodie.table.name': tableName, 'hoodie.datasource.write.partitionpath.field': 'city', 'hoodie.datasource.write.operation' : 'upsert', 'hoodie.datasource.meta.sync.enable' : 'true', }
inserts.write.format("hudi"). \
options(**hudi_options). \
mode("overwrite"). \
save(basePath)
columns2 = ["ts","uuid","rider","newcol","driver","fare","city"]
data2 =[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san_francisco", "newval")]
inserts2 = spark.createDataFrame(data2).toDF(*columns2)
inserts2.write.format("hudi"). \
options(**hudi_options). \
mode("append"). \
save(basePath)
```
Output - the values of new_col is wrong in the output and the values have shifted due to order of columns.
{+}-----------------{-}{{-}}{-}{-}{+}----------------{-}{{-}}+{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}------------------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}-----------{+}
|_hoodie_commit_time|_hoodie_commit_seqno| _hoodie_record_key|_hoodie_partition_path| _hoodie_file_name| ts| uuid| rider| driver| fare| newcol| city|
{+}-----------------{-}{{-}}{-}{-}{+}----------------{-}{{-}}+{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}------------------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}-----------{+}
| 20250116074123564|20250116074123564...|20250116074123564...| newval|439069ea-0289-4e5...|1695159649087|334e26e9-8355-45c...|rider-A| 19.1|san_francisco|driver-K| newval|
| 20250116074056916|20250116074056916...|20250116074056916...| san_francisco|af0b7568-fc25-4a3...|1695046462179|9909a8b1-2d15-4d3...|rider-D|driver-L| 33.9| null|san_francisco|
| 20250116074056916|20250116074056916...|20250116074056916...| san_francisco|af0b7568-fc25-4a3...|1695091554788|e96c4396-3fad-413...|rider-C|driver-M| 27.7| null|san_francisco|
| 20250116074056916|20250116074056916...|20250116074056916...| san_francisco|af0b7568-fc25-4a3...|1695159649087|334e26e9-8355-45c...|rider-A|driver-K| 19.1| null|san_francisco|
| 20250116074056916|20250116074056916...|20250116074056916...| sao_paulo|fc3f2a03-d722-4de...|1695516137016|e3cf430c-889d-401...|rider-F|driver-P| 34.15| null| sao_paulo|
| 20250116074056916|20250116074056916...|20250116074056916...| chennai|e2fd2aa4-4d78-433...|1695115999911|c8abbe79-8d89-47e...|rider-J|driver-T| 17.85| null| chennai|
{+}-----------------{-}{{-}}{-}{-}{+}----------------{-}{{-}}+{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}------------------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----------------{-}{{-}}{{-}}{-}---{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}---------{-}{{-}}{+}{+}{{-}}{-}----{-}{{-}}{{-}}{-}-----------{+}
Note- This fails with 0.X hudi version
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8871
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Data issue with hive sync if adding the column in between [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17364:
URL: https://github.com/apache/hudi/issues/17364#issuecomment-5351239598
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8871).
**Findings: needs the reproduction run against a current build with Hive sync enabled - I could not settle it statically.**
This one is worth prioritising because the report claims a **data** issue rather than only a schema mismatch, and it comes with a complete PySpark reproduction (write `trips_table`, add a column in the middle of the column list rather than appending, then read through Hive).
On `master` the sync path still funnels through `hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncTool.java:492`:
```java
syncClient.updateTableSchema(tableName, schema, schemaDiff);
```
Whether inserting a column mid-list produces a correct mapping in the Hive table depends on how `schemaDiff` is computed and on whether Hive resolves the underlying parquet columns by position or by name. That interaction is not decidable from the source, and getting it wrong in either direction would be a bad outcome to assert without evidence.
For whoever picks this up, the questions worth answering in order:
1. Does the Hive table's column list after sync match the Hudi schema's order, or does the new column land at the end?
2. If the order differs, do reads return values from the wrong column (data issue) or just fail (schema issue)?
3. Does it reproduce on both COW and MOR, and on both the snapshot and read-optimized Hive tables?
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org