[jira] [Commented] (HUDI-7229) Enable partial updates for CDC work payload

["Vinoth Chandar (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17843110#comment-17843110 ] 

Vinoth Chandar commented on HUDI-7229:
--------------------------------------

Punting this to 1.1 


 # [1.1] Implement support on top of data blocks.
 ## we need to pass change columns information and operation all the way to write handles, using a field in HoodieRecord
 ## ... 
 # [1.1] Implement support on top of cdc data blocks.
 ## we can track similar bitmaps for cdc data blocks as well
 ## we need to extend the new file group reader to also merge base and cdc blocks. (not just base and data blocks).

> Enable partial updates for CDC work payload
> -------------------------------------------
>
>                 Key: HUDI-7229
>                 URL: https://issues.apache.org/jira/browse/HUDI-7229
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Lin Liu
>            Assignee: Vinoth Chandar
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.1.0
>
>
> OLTP workloads on upstream databases, often update/delete/insert different columns in the table on each operation. Currently, Hudi can only supporting partial updates in cases where the same columns are being mutated in a given write to Hudi (e.g Spark SQL ETLs with MIT or Update statements). Here, we explore what it takes to support a smarter storage format, that can only encode the changed columns into log along with the different implementations.
> h2. Goals
>  # Enable partial update functionality for all existing and potential future CDC workloads without huge modification or duplication.
>  # Performance parity with current full-record updates or partial updates across the same set of columns
>  # Exhibit reduction in storage costs, by only storing the changed columns.
>  # Should also result in computation cost reductions by scanning/processing less data
>  # Should not affect the scalability of the existing system ingestion system. The number of files generated for partial update should not increase dramatically.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)