You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:47:49 UTC
[I] Enabling cols stats by default with writer for Spark enabled it for Flink [hudi]
hudi-bot opened a new issue, #16879:
URL: https://github.com/apache/hudi/issues/16879
[HUDI-8766] Enabling cols stats by default with writer (#12596)
- Enabling cols stats by default on writer in SPARK engine
- Added support for timestamp, Date, LocalDate, Decimal.
was expected to turn on column stats for Spark engine.
But I found performance decrease on this commit for Flink write due to enabled column stats.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9076
- Type: Bug
---
## Comments
25/Feb/25 10:07;geserdugarov;But there is corresponding task to enable it for Flink write by default:
[HUDI-8814] Enable col stats and partition stats by default on the writer with flink
https://issues.apache.org/jira/browse/HUDI-8814
Need to figure out which behavior is expected.;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enabling cols stats by default with writer for Spark enabled it for Flink [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16879:
URL: https://github.com/apache/hudi/issues/16879#issuecomment-4795645334
This appears to be fixed on master. The root cause was that `FlinkWriteClients.java` was calling `HoodieMetadataConfig.newBuilder()` without `.withEngineType(EngineType.FLINK)`, so the builder defaulted to `EngineType.SPARK` and turned on column stats by default for Flink writers — even though `HoodieMetadataConfig.Builder#getDefaultColStatsEnable` correctly returns `false` for FLINK/JAVA.
Fix: commit [c9cf9d5f365a](https://github.com/apache/hudi/commit/c9cf9d5f365a14997ba49c96d975aa29bf48773c) (HUDI-9175, PR #12977, Mar 18 2025) added `.withEngineType(EngineType.FLINK)` to the metadata config builder in `FlinkWriteClients.java` (now at line 235, with the comment `// this affects the default value inference`).
With the engine type wired through, `getDefaultColStatsEnable(FLINK)` returns `false` and col stats is no longer enabled by default for Flink writers. Closing as resolved unless you can reproduce on a recent master/release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Enabling cols stats by default with writer for Spark enabled it for Flink [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16879: Enabling cols stats by default with writer for Spark enabled it for Flink
URL: https://github.com/apache/hudi/issues/16879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org