You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/07/31 08:11:37 UTC

[I] [Spark][LSM] Support standard write operations and compaction [hudi]

cshuo opened a new issue, #19437:
URL: https://github.com/apache/hudi/issues/19437

   ### Task Description
   
   **What needs to be done:**
   
   Support the standard Spark write operations and MOR compaction services for LSM-layout data tables.
   
   **Why this task is needed:**
   
   Real Spark workloads mix insert, upsert, delete, and overwrite operations. Restricting an LSM table to upsert-only behavior prevents normal ingestion workflows such as an initial insert followed by incremental upserts.
   
   ### Implementation Scope
   
   - Support the HoodieRecord/RDD paths for:
     - `UPSERT` and `UPSERT_PREPPED`
     - `INSERT` and `INSERT_PREPPED`
     - `DELETE` and `DELETE_PREPPED`
     - `INSERT_OVERWRITE` and `INSERT_OVERWRITE_TABLE`
     - `DELETE_PARTITION`
   - Sort LSM input in `BaseSparkCommitActionExecutor` using the comparator selected from the base-file format.
   - Remove the Spark LSM rejection for insert and insert-prepped while retaining the metadata-table exemption.
   - Support MOR `COMPACT` and `LOG_COMPACT`, producing sorted base files and native log blocks.
   - Keep replace-commit, file-group replacement, rollback, clean, and index-update semantics unchanged.
   - Define duplicate-key behavior explicitly:
     - A base-file-only insert may preserve duplicate physical keys.
     - After an upsert/delete introduces a merge path, equal keys are merged as one logical key.
   
   ### Acceptance Criteria
   
   - [ ] All listed operations succeed for applicable COW and MOR LSM tables.
   - [ ] Every newly written base file is ordered according to its base-file format.
   - [ ] An `INSERT -> UPSERT` MOR workflow returns the latest logical records.
   - [ ] A duplicate-key insert is preserved on the base-only path and converges through a later LSM merge.
   - [ ] Regular and log compaction preserve snapshot correctness and sorted-run invariants.
   - [ ] Replace commit and file-group replacement metadata remain correct.
   - [ ] Targeted Spark client/DataSource tests, compile, checkstyle/scalastyle, and `git diff --check` pass.
   
   ### Related Issues
   
   **Parent feature issue:** #19065
   
   **Depends on:** #19436
   
   **Out of scope:** bulk insert, bucket-specific paths, clustering, bootstrap, and metadata-table indexing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Spark][LSM] Support standard write operations and compaction [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #19437: [Spark][LSM] Support standard write operations and compaction
URL: https://github.com/apache/hudi/issues/19437


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org