You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2026/09/02 04:31:13 UTC
[PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
rangareddy opened a new pull request, #19815:
URL: https://github.com/apache/hudi/pull/19815
### Describe the issue this Pull Request addresses
Relates to https://issues.apache.org/jira/browse/HUDI-8592 (#17302), which reports that `hoodie.parquet.outputtimestamptype=TIMESTAMP_MILLIS` is ignored and the Parquet file is always written with `timestamp-micros`.
The config's own description is a large part of why this gets reported. It currently reads:
> Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.
That promises the config governs Spark writes generally. It does not. `HoodieRowParquetWriteSupport` is its only consumer, copying the value into `spark.sql.parquet.outputTimestampType` on the hadoop conf it builds, which covers the Spark row-writer Parquet path. Writes that go through the Avro path instead, such as `insert` and `upsert`, get their Parquet timestamp type from the table's Avro schema via parquet-avro, so the config has no effect on them. That matches the observation in the ticket that the setting appears to work under `BULK_INSERT` and not otherwise.
This PR corrects the description only. It does **not** make the config effective on the Avro path: doing that means changing the timestamp logical type in the table's Avro schema, which is a storage-format decision and wants its own discussion rather than a drive-by change. HUDI-8592 should stay open for that.
### Summary and Changelog
Users reading the config table on the website (and in the IDE) now see which write paths `hoodie.parquet.outputtimestamptype` actually governs, instead of a description that implies all Spark writes.
- `HoodieStorageConfig.PARQUET_OUTPUT_TIMESTAMP_TYPE`: narrowed `withDocumentation(...)` to state that it applies to the Spark row-writer Parquet path (for example bulk insert), and that Avro-path writes derive the Parquet timestamp type from the table's Avro schema and are unaffected.
No code was copied.
### Impact
No public API or behavior change. The config key, its default (`TIMESTAMP_MICROS`), and its `markAdvanced()` status are untouched, and no code path reads the description. The generated config table on the Hudi website will show the new text.
Worth noting for a follow-up, not changed here: `HoodieWriteConfig.parquetOutputTimestampType()` has no callers anywhere in the repo. Removing it is a public method removal, so it does not belong in a documentation change.
### Risk Level
none
Documentation string only. Verified with `mvn compile checkstyle:check -pl hudi-common`, which passes.
### Documentation Update
This PR is the documentation update. The config description is corrected in place, which is what feeds the generated config tables on the Hudi website, so no separate website change is needed.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3921560755
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
Review Comment:
Done, both dropped in this PR rather than a follow-up: the dead `hadoopConf.set` at L154 and `HoodieWriteConfig.parquetOutputTimestampType()`. The builder setter `HoodieStorageConfig.Builder.parquetOutputTimestampType` stays, since it is still how a caller sets the (now deprecated) key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3933395819
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
Review Comment:
Applied in 9a3ea36362b0. `Builder.parquetOutputTimestampType()` now carries `@Deprecated` plus a javadoc pointing at the writer schema, following `HoodieWriteConfig.Builder.withPopulateMetaFields`. It was the last undeprecated surface for this config, and the write side at that: the accessor has no callers in the tree, whereas the setter is what a programmatic caller actually invokes.
The one cross-class caller is the new test that pins the no-op deliberately, so it takes `@SuppressWarnings("deprecation")`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3921560612
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
+ + "Writes that go through the Avro path, such as insert and upsert, derive the Parquet timestamp type from the table's "
+ + "Avro schema and are unaffected by this config.");
Review Comment:
Fixed, and thanks for the probe: I confirmed it independently before changing anything. `convertField` (L798-816) takes the unit from `resolvedSchema`s `TimePrecision` and `spark.sql.parquet.outputTimestampType` appears in that class only at L154 where it is set, never read. The description now says no effect since 1.1.0 and points at the writer schema, and the config carries `@Deprecated` plus `deprecatedAfter("1.1.0")`.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
Review Comment:
Done, both dropped in this PR rather than a follow-up: the dead `hadoopConf.set` at L154 and `HoodieWriteConfig.parquetOutputTimestampType()`. The builder setter `HoodieStorageConfig.Builder.parquetOutputTimestampType` stays, since it is still how a caller sets the (now deprecated) key.
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
+ + "Writes that go through the Avro path, such as insert and upsert, derive the Parquet timestamp type from the table's "
+ + "Avro schema and are unaffected by this config.");
Review Comment:
Added, but in `TestHoodieInternalRowParquetWriter` rather than `TestHoodieRowParquetWriteSupport`: the latter cannot construct the write support because it needs a Spark adapter that is not on the `hudi-spark-client` test classpath, as its own class javadoc records. Both directions are pinned, `TIMESTAMP_MILLIS` config against a timestamp-micros schema gives MICROS and the default against a timestamp-millis schema gives MILLIS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5504513059
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81656740e9c0766529bd60044a9fcf881315f5c5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5525153734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
}, {
"hash" : "d39615fa4ddd7da929381165d1674bc29e598634",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16800",
"triggerID" : "d39615fa4ddd7da929381165d1674bc29e598634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d39615fa4ddd7da929381165d1674bc29e598634 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3921656686
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
Review Comment:
π€ **Line 657:** Now that the config is `@Deprecated`/no-op, would it be worth also marking the fluent `Builder.parquetOutputTimestampType(...)` setter `@Deprecated`? A user calling it via the builder gets a silent no-op with no deprecation signal (they never see the config's doc string), which is the same confusion this PR is trying to remove.
<sub><i>β οΈ AI-generated; verify before applying. React π/π to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5522517028
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5504467944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81656740e9c0766529bd60044a9fcf881315f5c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5521349661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81656740e9c0766529bd60044a9fcf881315f5c5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759)
* fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5523893509
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
}, {
"hash" : "d39615fa4ddd7da929381165d1674bc29e598634",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d39615fa4ddd7da929381165d1674bc29e598634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794)
* d39615fa4ddd7da929381165d1674bc29e598634 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5505495914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81656740e9c0766529bd60044a9fcf881315f5c5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3923201959
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
Review Comment:
Correction to my earlier reply on this thread: I dropped the dead `hadoopConf.set` as you asked, but I have kept `HoodieWriteConfig.parquetOutputTimestampType()` and marked it `@Deprecated` rather than deleting it. It has no callers in the tree, but it is public, and `HoodieWriteConfig` deprecates public members rather than removing them (76 `@Deprecated` markers in that class), so deleting it would be source-incompatible for downstream integrations with no benefit. The javadoc says the returned value does not describe what gets written, which I think serves the same purpose you were after. Happy to delete it instead if you would rather.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5523968141
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
}, {
"hash" : "d39615fa4ddd7da929381165d1674bc29e598634",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16800",
"triggerID" : "d39615fa4ddd7da929381165d1674bc29e598634",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794)
* d39615fa4ddd7da929381165d1674bc29e598634 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16800)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(config): deprecate hoodie.parquet.outputtimestamptype, a no-op since 1.1.0 [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19815:
URL: https://github.com/apache/hudi/pull/19815
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5504517972
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19815?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 78.31%. Comparing base ([`a2788ee`](https://app.codecov.io/gh/apache/hudi/commit/a2788eef922e2375432ead3fe50af7fe5f55faf7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`8165674`](https://app.codecov.io/gh/apache/hudi/commit/81656740e9c0766529bd60044a9fcf881315f5c5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19815 +/- ##
============================================
- Coverage 78.31% 78.31% -0.01%
Complexity 33887 33887
============================================
Files 2541 2541
Lines 141728 141728
Branches 17182 17182
============================================
- Hits 110993 110991 -2
- Misses 23031 23032 +1
- Partials 7704 7705 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.65% <100.00%> (ΓΈ)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.20% <ΓΈ> (-0.01%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.65% <ΓΈ> (ΓΈ)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.21% <ΓΈ> (ΓΈ)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.57% <ΓΈ> (ΓΈ)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.06% <ΓΈ> (ΓΈ)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.71% <ΓΈ> (ΓΈ)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.56% <ΓΈ> (ΓΈ)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.88% <ΓΈ> (ΓΈ)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ΓΈ> (ΓΈ)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ΓΈ> (ΓΈ)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19815/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.96% <ΓΈ> (ΓΈ)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19815/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19815/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.07% <100.00%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19815?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [...apache/hudi/common/config/HoodieStorageConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19815?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Fconfig%2FHoodieStorageConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9Ib29kaWVTdG9yYWdlQ29uZmlnLmphdmE=) | `90.25% <100.00%> (ΓΈ)` | |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19815/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19815:
URL: https://github.com/apache/hudi/pull/19815#discussion_r3920941197
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
+ + "Writes that go through the Avro path, such as insert and upsert, derive the Parquet timestamp type from the table's "
+ + "Avro schema and are unaffected by this config.");
Review Comment:
**blocker:** On master the row-writer path does not honor this config either, so the narrowed description is still incorrect. Since #13882 (4d95b2c2d165, in 1.1.0) `HoodieRowParquetWriteSupport` builds the Parquet schema itself and takes MICROS/MILLIS from the writer schema's logical type (`convertField`, L800-816); the value copied into the hadoop conf at L154 is never read. Verified with a direct write against current snapshot jars: `TIMESTAMP_MILLIS` with a `timestamp-micros` schema gives `TIMESTAMP(MICROS)`, and the default with a `timestamp-millis` schema gives `TIMESTAMP(MILLIS)`. Could we document it as a no-op since 1.1.0 and mark it `.deprecatedAfter("1.1.0")`, pointing at the writer schema instead?
```suggestion
.withDocumentation("No effect since 1.1.0. Both the Spark row writer and the Avro Parquet writer derive the "
+ "Parquet timestamp unit from the writer schema's logical type (timestamp-micros or timestamp-millis); "
+ "declare the precision in the writer schema (for example via hoodie.write.schema) instead.")
.deprecatedAfter("1.1.0");
```
<details>
<summary>Probe results and why bulk insert used to work</summary>
| config | writer schema | parquet footer |
|---|---|---|
| `TIMESTAMP_MILLIS` | timestamp-micros | `INT64 TIMESTAMP(MICROS,true)` |
| `TIMESTAMP_MICROS` (default) | timestamp-millis | `INT64 TIMESTAMP(MILLIS,true)` |
| `INT96` | timestamp-micros | `INT64 TIMESTAMP(MICROS,true)` |
| unset | timestamp-micros | `INT64 TIMESTAMP(MICROS,true)` |
Each row constructs `HoodieRowParquetWriteSupport` from a `HoodieConfig` holding `hoodie.avro.schema` plus the config value, writes one `InternalRow` through `ParquetWriter`, and reads the footer back. Before #13882 the class extended Spark's `ParquetWriteSupport`, whose `init()` reads `spark.sql.parquet.outputTimestampType`; that is why bulk insert honored the config through 1.0.x and the HUDI-8592 reporter saw it work there. Since 1.1.0 it extends parquet's `WriteSupport` directly, so the key is set but never read on any path.
</details>
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
Review Comment:
**minor:** Not blocking. `HoodieRowParquetWriteSupport.java:154` still does `hadoopConf.set("spark.sql.parquet.outputTimestampType", ...)` even though nothing has read it since #13882, and that line is what makes this description (and the HUDI-8592 triage) look right on a code read. `HoodieWriteConfig.parquetOutputTimestampType()` has no callers either, as the PR body notes. Would it be worth dropping both in this PR rather than a follow-up, so the next reader does not reach the same conclusion?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -240,7 +240,9 @@ public class HoodieStorageConfig extends HoodieConfig {
.key("hoodie.parquet.outputtimestamptype")
.defaultValue("TIMESTAMP_MICROS")
.markAdvanced()
- .withDocumentation("Sets spark.sql.parquet.outputTimestampType. Parquet timestamp type to use when Spark writes data to Parquet files.");
+ .withDocumentation("Sets spark.sql.parquet.outputTimestampType for the Spark row-writer Parquet path (for example bulk insert). "
+ + "Writes that go through the Avro path, such as insert and upsert, derive the Parquet timestamp type from the table's "
+ + "Avro schema and are unaffected by this config.");
Review Comment:
**minor:** Not blocking. Nothing under any `src/test` references `hoodie.parquet.outputtimestamptype` or `PARQUET_OUTPUT_TIMESTAMP_TYPE`, which is how the description drifted from the behaviour unnoticed. Could we add a two-case assertion to `TestHoodieRowParquetWriteSupport`: config `TIMESTAMP_MILLIS` with a `timestamp-micros` writer schema yields a `TIMESTAMP(MICROS)` footer, and the default with a `timestamp-millis` schema yields `TIMESTAMP(MILLIS)`? That pins whatever the description ends up saying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs(config): scope hoodie.parquet.outputtimestamptype to the row-writer path [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19815:
URL: https://github.com/apache/hudi/pull/19815#issuecomment-5521405588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759",
"triggerID" : "81656740e9c0766529bd60044a9fcf881315f5c5",
"triggerType" : "PUSH"
}, {
"hash" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794",
"triggerID" : "fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 81656740e9c0766529bd60044a9fcf881315f5c5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16759)
* fc6f23d2a1cef3a2797bc5f4e0715626bdd7cfcd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16794)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org