You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/11/06 08:02:49 UTC
[GitHub] [hudi] satishkotha commented on pull request #2196: [HUDI-1349]spark sql support overwrite use replace action
satishkotha commented on pull request #2196:
URL: https://github.com/apache/hudi/pull/2196#issuecomment-722937587
> @satishkotha
> A. Thanks so much. This pr need to solved the issue with better approach.
> Now I am more clear about overwrite semantic between table.overwrite and spark sql overwrite for hudi.
>
> B. Also spark sql for hudi overwrite should have the ability just like spark sql 、hive 、 delta lake.
> these engine have three mode for overwrite about partition:
>
> 1. Dynamic Partition : delete all partition data ,and the insert the new data for different
> 2. Static partition: just overwrite the partition which is user specified
> 3. Mixed partition: mixed of 1 and 2
> more detail in :
> https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-dml-insert-overwrite-table.html
> https://www.programmersought.com/article/47155360487/
>
> C. our plan
>
> 1. Now spark sql for hudi overwrite is Dynamic Partition. I will resolved it in this issue HUDI-1349, first support delete all partition in [HUDI-1350](https://issues.apache.org/jira/browse/HUDI-1350) , then land this issue. (Just like @satishkotha 's Suggestion)
> 2. Now spark sql for hudi does not support "Static partition" mode, will then land it in [HUDI-1374](https://issues.apache.org/jira/browse/HUDI-1374)
> 3. future support "Mixed partition" mode
>
> cc @n3nash @vinothchandar please help to review if the plan about spark sql for hudi overwirte is suitable.
> It is also possible that my understanding is inappropriate
Just fyi, in the [RFC](https://cwiki.apache.org/confluence/display/HUDI/RFC+-+18+Insert+Overwrite+API#RFC18InsertOverwriteAPI-API) we discussed having 'insert_overwrite_table' operation to support dynamic partitioning. static partitioning is supported by 'insert_overwrite'.
But, agree with getting approval from Nishith/Vinoth before starting implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org