You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:09:14 UTC
[I] add new config controls whether input rdd should be first persist before insert. [hudi]
hudi-bot opened a new issue, #15596:
URL: https://github.com/apache/hudi/issues/15596
When our input data comes from a complex rdd lineage, hudi writing will lead to repeated calculations.
For example, we will de duplicate according to the key of the input data, and we will obtain all partitions to be written to the data in the tag location. So I think we should cache the data to be written for downstream use.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5284
- Type: New Feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add new config controls whether input rdd should be first persist before insert. [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15596: add new config controls whether input rdd should be first persist before insert.
URL: https://github.com/apache/hudi/issues/15596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add new config controls whether input rdd should be first persist before insert. [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15596:
URL: https://github.com/apache/hudi/issues/15596#issuecomment-3630120973
### Linked PR(s)
- https://github.com/apache/hudi/pull/7309
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] add new config controls whether input rdd should be first persist before insert. [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15596:
URL: https://github.com/apache/hudi/issues/15596#issuecomment-4795451406
Triage on current master: this appears to be addressed.
In `BaseSparkCommitActionExecutor.execute()` (`hudi-client/hudi-spark-client/.../BaseSparkCommitActionExecutor.java`, lines ~174-180), Hudi already persists the tagged input RDD before building the workload profile and performing the actual writes, so the dedup + tag-location lineage isn't recomputed by the downstream stages. The storage level is configurable via `hoodie.write.tagged.record.storage.level` (default `MEMORY_AND_DISK_SER`), defined in `HoodieWriteConfig.TAGGED_RECORD_STORAGE_LEVEL_VALUE`. The persistence is also skipped if the caller has already cached the source RDD upstream (`shouldPersistInputRecords`, line ~222).
This was introduced by HUDI-5786 / PR #7941 (commit `be92be65`). If your scenario still shows recomputation, please share a small repro and we can reopen.
Closing as resolved — feel free to reopen with details if this doesn't match your case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org