You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:11:05 UTC
[I] Support Prepped operations w/ spark-sql [hudi]
hudi-bot opened a new issue, #15613:
URL: https://github.com/apache/hudi/issues/15613
If incoming data was consumed from a hudi table, it already has the fileName from which we can glean the fileId info. And adding a prepped flow, we can avoid the index lookup altogether.
We can add this support DELETEs in spark-sql. But may be worth considering for UPDATE and MERGE INTO as well.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5337
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Prepped operations w/ spark-sql [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15613:
URL: https://github.com/apache/hudi/issues/15613#issuecomment-4795454288
Triaging on current master: this appears to be addressed under [HUDI-6315](https://issues.apache.org/jira/browse/HUDI-6315), which superseded HUDI-5337.
- **DELETE** (spark-sql): prepped flow implemented in #8978 — `DeleteHoodieTableCommand` sets `SPARK_SQL_WRITES_PREPPED_KEY=true` when `hoodie.spark.sql.optimized.writes.enable=true` (default `true` since 0.14.0), bypassing key generation and index lookup via `HoodieSparkSqlWriter` (see `HoodieSparkSqlWriter.scala` lines 255-427).
- **UPDATE** (spark-sql): same path, implemented in #8956 (`UpdateHoodieTableCommand.scala`).
- **MERGE INTO**: `HoodieWriteConfig.SPARK_SQL_MERGE_INTO_PREPPED_KEY` exists and is wired through `HoodieSparkSqlWriter`, but `MergeIntoHoodieTableCommand` currently auto-enables it only for primary-keyless tables (`isPrimaryKeylessTable.toString` at line 834). If broader MERGE INTO support is still desired (pk tables when source carries meta fields), it would be worth filing a follow-up JIRA scoped to that case.
Given DELETE and UPDATE — the explicit asks in this issue — are supported and on by default, suggest closing this in favor of a narrower follow-up for MERGE INTO if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support Prepped operations w/ spark-sql [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15613: Support Prepped operations w/ spark-sql
URL: https://github.com/apache/hudi/issues/15613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org