You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:36:59 UTC
[I] Hudi is not honoured hoodie.parquet.outputtimestamptype value as TIMESTAMP_MILLIS [hudi]
hudi-bot opened a new issue, #17302:
URL: https://github.com/apache/hudi/issues/17302
{*}Problem Statement{*}:
While writing data to a Hudi table with the {{hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS}} Hudi option, the setting is not honored, and the Parquet file always defaults to {{{}timestamp-micros{}}}.
{*}Solution{*}:
Hudi should honor the {{hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS}} setting when specified in the Hudi options.
*Reproducible Code:*
For more details, refer the following hudi issue.
{{[https://github.com/apache/hudi/issues/12339]}}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8592
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9113
- Fix version(s):
- 1.1.0
---
## Comments
05/Feb/25 05:58;ktblsva;looks like it works for BULK_INSERT mode
{code:java}
val name = this.getClass.getSimpleName.replace("$", "")
val sparkConf = new SparkConf().setAppName(name).setIfMissing("spark.master", "local[2]")
val spark = SparkSession.builder.appName(name).config(sparkConf)
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension")
.config("spark.sql.hive.convertMetastoreParquet", "false")
.getOrCreate()
val tableName = name
val basePath = f"file:///tmp/warehouse/$tableName"
val schema = StructType(Array(
StructField("field1", IntegerType, nullable = false),
StructField("field2", StringType, nullable = true),
StructField("field3", TimestampType, nullable = false)
))
val data = Seq(
Row(1, "A", java.sql.Timestamp.valueOf("2023-10-01 10:00:00.540040")),
Row(2, "B", java.sql.Timestamp.valueOf("2023-10-01 11:30:00.240030")),
Row(3, "C", java.sql.Timestamp.valueOf("2023-10-01 12:45:00.140022"))
)
val df = spark.createDataFrame(spark.sparkContext.parallelize(data), schema)
// Hudi write options
val hudiOptions = Map(
"hoodie.table.name" -> tableName,
"hoodie.datasource.write.recordkey.field" -> "field1",
"hoodie.datasource.write.precombine.field" -> "field2",
"hoodie.parquet.outputtimestamptype" -> "TIMESTAMP_MILLIS",
DataSourceWriteOptions.OPERATION.key -> DataSourceWriteOptions.BULK_INSERT_OPERATION_OPT_VAL,
//"hoodie.datasource.write.keygenerator.consistent.logical.timestamp.enabled" -> "true"
)
// Write the DataFrame to Hudi
df.write.format("hudi").options(hudiOptions).mode("overwrite").save(basePath)
spark.stop() {code};;;
---
13/Feb/25 15:11;rangareddy.avula@gmail.com;Hi [~ktblsva]
We need to make sure, it needs to work for any writer operation.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi is not honoured hoodie.parquet.outputtimestamptype value as TIMESTAMP_MILLIS [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on issue #17302:
URL: https://github.com/apache/hudi/issues/17302#issuecomment-5539095717
`hoodie.parquet.outputtimestamptype` has had no effect on any Spark write path since 1.1.0, bulk insert included.
The Feb 2025 comment above ("looks like it works for BULK_INSERT mode") was accurate then, on 1.0.x. It stopped being true in #13882 (`4d95b2c2d165`, first released in 1.1.0), which changed `HoodieRowParquetWriteSupport` from `extends ParquetWriteSupport` (Spark's, whose `init()` reads `spark.sql.parquet.outputTimestampType` off the Hadoop conf) to `extends WriteSupport<InternalRow>` with its own `init()`. The constructor kept setting that Hadoop conf key, but nothing reads it any more, so the value is inert.
Running the repro from this issue against current master with `hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS` and reading the Parquet footer back:
| write path | `field3` in the footer |
| --- | --- |
| `bulk_insert` (row writer) | `TIMESTAMP(MICROS,true)` |
| `bulk_insert` with `hoodie.datasource.write.row.writer.enable=false` | `TIMESTAMP(MICROS,true)` |
| `insert` | `TIMESTAMP(MICROS,true)` |
| `upsert` | `TIMESTAMP(MICROS,true)` |
Setting the Spark session conf `spark.sql.parquet.outputTimestampType=TIMESTAMP_MILLIS` is ignored on the same paths.
Both the Spark row writer and the Avro Parquet writer now take the timestamp unit from the writer schema's logical type, so millis has to be declared there:
```scala
.option("hoodie.write.schema",
"""{"type":"record","name":"rec","fields":[
{"name":"field1","type":"int"},
{"name":"field2","type":["null","string"],"default":null},
{"name":"field3","type":{"type":"long","logicalType":"timestamp-millis"}}]}""")
```
That gives `TIMESTAMP(MILLIS,true)` on bulk insert.
#19815 documents the config as a no-op, marks it `@Deprecated` with `deprecatedAfter("1.1.0")`, and points at the writer schema. If that is the resolution here, this issue can be closed once it merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi is not honoured hoodie.parquet.outputtimestamptype value as TIMESTAMP_MILLIS [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17302:
URL: https://github.com/apache/hudi/issues/17302#issuecomment-5365818970
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8592).
**Findings: needs a reproduction against a current build before it can be dispositioned.**
The report is concrete and actionable as written: setting hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS is ignored and the parquet file is always written with timestamp-micros.
I have deliberately not called this one from the source. The surrounding code has been substantially rewritten since it was filed - table versions 8 and 9, the `BufferedRecordMerger` family replacing payload-based merging, the file-group-reader migration, and the `hudi-common` package reorganisations in #19193 / #19195. With that much churn, inferring either "still broken" or "silently fixed" would be a guess rather than a finding, and both errors are costly here.
The good news is that reproducing it is cheap: write a table with that option set and inspect the resulting parquet schema with parquet-tools or a schema read; no cluster needed.
There is also no commit anywhere in the tree referencing HUDI-8592, so if this was fixed it was incidental to another change rather than deliberate - which is another reason to verify rather than assume.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org