You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 09:46:12 UTC
[I] Date columns showing incorrect integer values while doing CDC read [hudi]
hudi-bot opened a new issue, #16408:
URL: https://github.com/apache/hudi/issues/16408
Date columns are showing integer values while doing CDC read.
For timestamp columns also it is converting into epoch , may be doing similar conversion for date column causing wrong integer values.
Reproducible Code -
```
columns = ["ts","uuid","rider","driver","fare","city", "report_date"]
data =[(1695159649087,"334e26e9-8355-45cc-97c6-c31daf0df330","rider-A","driver-K",19.10,"san_francisco", "2022-01-01"),
(1695091554788,"e96c4396-3fad-413a-a942-4cb36106d721","rider-B","driver-L",27.70 ,"san_francisco", "2022-01-01"),
(1695046462179,"9909a8b1-2d15-4d3d-8ec9-efc48c536a00","rider-C","driver-M",33.90 ,"san_francisco", "2022-01-01"),
(1695516137016,"e3cf430c-889d-4015-bc98-59bdce1e530c","rider-C","driver-N",34.15,"sao_paulo", "2022-01-01")]
inserts = spark.createDataFrame(data).toDF(*columns)
hudi_options =
{ 'hoodie.table.name': tableName, 'hoodie.datasource.write.recordkey.field' : 'uuid', 'hoodie.datasource.write.precombine.field' : 'ts', 'hoodie.datasource.write.partitionpath.field': 'city', 'hoodie.datasource.write.reconcile.schema':'true', 'hoodie.table.cdc.enabled':'true', }
inserts = inserts.withColumn("report_date", expr("CAST(report_date as timestamp)"))
inserts.write.format("hudi"). \
options(**hudi_options). \
mode("overwrite"). \
save(basePath)
cdc_read_options = \{ 'hoodie.datasource.query.incremental.format': 'cdc', 'hoodie.datasource.query.type': 'incremental', 'hoodie.datasource.read.begin.instanttime': 0 }
df = spark.read.format("hudi").load(basePath)
df.show(10, False)
spark.read.format("hudi"). \
options(**cdc_read_options). \
load(basePath).show(10, False)
```
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7461
- Type: Bug
- Fix version(s):
- 0.16.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Date columns showing incorrect integer values while doing CDC read [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #16408:
URL: https://github.com/apache/hudi/issues/16408#issuecomment-5056575188
This issue was reviewed as part of the JIRA-migrated backlog triage.
Confirmed reproducible on current master (this is NOT fixed; JIRA HUDI-7461 is still Open). Root cause: CDC before/after JSON images are built by `InternalRowToJsonStringConverter` (hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/InternalRowToJsonStringConverter.scala). Its `convertField` handles `String`/`Array`/`Map`/`Struct` explicitly but routes all other types — including `DateType` and `TimestampType` — through the `case _ => value` fallthrough. That returns the raw `InternalRow` storage (epoch-days `Int` for date, epoch-micros `Long` for timestamp) instead of a formatted value, so date/timestamp columns appear as integers in the CDC read output. There is no date/timestamp coverage in `TestInternalRowToJsonStringConverter`.
Fix scope is a contained change in that converter (add `DateType`/`TimestampType` cases that render proper values) plus regression tests. Tracking as `stream:fix` / incr-processing (read path). Keeping open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org