[I] [Discussion WIP] Allow replace of an entire partition for improved performance for reading/cleaning dataset after a delete_partition [hudi]

["kbuci (via GitHub)" <gi...@apache.org>]

kbuci opened a new issue, #19428:
URL: https://github.com/apache/hudi/issues/19428

   Currently, the HUDI delete_partition will replace an entire partition by finding all file ids in each target partition, and marking them as replaced. Later on when this replacecommit instant is eligible for clean, the clean table service will find all these replaced file ids and will delete them from DFS and remove their entried from MDT partitions (like FILES). This is correct and the same with how clustering/insert_overwrite replacecommits are handled. 
   We have seen cases where, if we perform a delete_partition on many large partitions (large meaning 10,000+), then the subsequent clean will need more spark resources (since it has to process many file entries). For append-only datasets where we just insert files in new partitions and use delete_partition to delete old expired TTLed partitions, it would be ideal if we could avoid this cost for `clean`ing. 
   As an optimization, could we update HUDI such that we could mark an entire partition as "replaced"? The idea is that all relevant flows that read instants for replaced file ids would be updated to avoiding needing to read/process  each individual file id entry in a replaced partition.
   - The filesystem view APIs would return 0 file ids for a partition if it sees that the partition is marked as replaced in a replacecommit and no later instant has added new file ids to the partition (instead of having to update in the internal mapping of `partition -> [replace file id list]`)
   - Clean planner would not attempt to find each file id in a replaced partition - it can create a smaller set of each partition that has been marked for replace and hasn't had a later instant writing new files to it. Then the clean execution can issue DFS delete call for all files in the partition. And remove all related records in MDT partitions 
   
   
   cc @nsivabalan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Discussion WIP] Allow replace of an entire partition for improved performance for reading/cleaning dataset after a delete_partition [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #19428:
URL: https://github.com/apache/hudi/issues/19428#issuecomment-5172403139

   We added support to max the instants to clean up in one round of cleaning right. would that config be of any use? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Discussion WIP] Allow replace of an entire partition for improved performance for reading/cleaning dataset after a delete_partition [hudi]

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 commented on issue #19428:
URL: https://github.com/apache/hudi/issues/19428#issuecomment-5173375635

   Spark `DROP PARTITION` has the purge syntax. As of Apache Hudi 1.2.0 and current `master` (August 2026), Spark SQL `PURGE` does **not** immediately delete a Hudi partition’s physical files.
   
   ```sql
   ALTER TABLE hudi_table
   DROP PARTITION (dt = '2026-08-01')
   PURGE;
   ```
   
   Spark accepts the `[PURGE]` syntax, but Hudi’s implementation receives the `purge` flag and never uses it. Instead, it records a `DELETE_PARTITION` operation and explicitly relies on **lazy cleaning**. The partition disappears from the current Hudi snapshot immediately, while its files remain until the cleaner removes them. See the [[current Hudi command implementation](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableDropPartitionCommand.scala)](https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableDropPartitionCommand.scala), [[Hudi SQL DDL](https://hudi.apache.org/docs/sql_ddl/)](https://hudi.apache.org/docs/sql_ddl/), and [[Spark’s generic syntax](https://spark.apache.org/docs/4.0.0/sql-ref-syntax-ddl-alter-table.html)](https://spark.apache.org/docs/4.0.0/sql-ref-syntax-ddl-alter-tab
 le.html).
   
   To reclaim storage promptly, run cleaning explicitly—for example:
   
   ```sql
   CALL run_clean(
     table => 'database.hudi_table',
     retain_commits => 1
   );
   ```
   
   Lowering retention reduces rollback/time-travel protection, so use it carefully. Don’t manually remove the partition directory, because that can leave Hudi’s timeline and metadata inconsistent.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org