[jira] [Updated] (HUDI-6478) Simplify INSERT_INTO configs

["Raymond Xu (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-6478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raymond Xu updated HUDI-6478:
-----------------------------
    Fix Version/s: 0.14.0

> Simplify INSERT_INTO configs
> ----------------------------
>
>                 Key: HUDI-6478
>                 URL: https://issues.apache.org/jira/browse/HUDI-6478
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: spark-sql
>            Reporter: sivabalan narayanan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.14.0
>
>
> We have 2 to 3 diff configs in the mix for INSERT_INTO command. lets try to simplify them.
>  
> hoodie.sql.insert.mode, drop dups, hoodie.sql.bulk.insert.enable and datasource.operation.type.
>  
> Rough notes:
>  
> hoodie.sql.bulk.insert.enable: true | false.
>  
> hoodie.sql.insert.mode: STRICT| NON_STRICT | UPSERT
> STRICT: we can't re-ingest same record again. will throw if found duplicates to be ingested again.
> NON_STRICT: no such constraints. but has to be set along w/ bulk_insert(if its enabled). if not, exception will be thrown.
> UPSERT: default insert.mode(until a week back where in we switch to make bulk_insert the default for INSERT_INTO). will take care of de-dup. will use OverwriteWithLatestAvroPayload(which means that we can update an existing record across batches).
>  
> datasource.operation.type: insert, bulk_insert, upsert
>  
> drop.dups: Drop new incoming records if it already exists.
>  
> Proposal:
>  
>  * We will introduce a new config named "hoodie.sql.write.operation" which will have 3 values ("insert", "bulk_insert" and "upsert"). Default value will be "insert" for INSERT_INTO.
>  ** Deprecate hoodie.sql.insert.mode and "hoodie.sql.bulk.insert.enable".
>  * Also, enable "hoodie.merge.allow.duplicate.on.inserts" = true if operation type is "Insert" for both spark-sql and spark-ds. This will maintain duplicates but still help w/ small file management with "insert"s.
>  * Introduce a new config named "hoodie.datasource.insert.dedupe.policy" whose valid values are "ignore, fail and drop". Make "ignore" as default. "fail" will mimic "STRICT" mode we support as of now. Even spark-ds users can use the fail/STRICT behavior if need be.
>  ** Deprecate hoodie.datasource.insert.drop.dups.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)