You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Vinoth Chandar (Jira)" <ji...@apache.org> on 2021/01/26 00:21:03 UTC
[jira] [Resolved] (HUDI-1153) Spark DataSource and Streaming Write
must fail when operation type is misconfigured
[ https://issues.apache.org/jira/browse/HUDI-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinoth Chandar resolved HUDI-1153.
----------------------------------
Resolution: Fixed
> Spark DataSource and Streaming Write must fail when operation type is misconfigured
> -----------------------------------------------------------------------------------
>
> Key: HUDI-1153
> URL: https://issues.apache.org/jira/browse/HUDI-1153
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Spark Integration
> Reporter: Balaji Varadarajan
> Assignee: Sreeram Ramji
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.7.0
>
>
> Context: [https://github.com/apache/hudi/issues/1902#issuecomment-669698259]
>
> If you look at DataSourceUtils.java, [https://github.com/apache/hudi/blob/master/hudi-spark/src/main/java/org/apache/hudi/DataSourceUtils.java#L257]
> we are string comparison to determine operation type which is a bad idea and a typo could result in "upsert" being used silently.
>
> Just like [https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L187] being used for DeltaStreamer, we need similar enums defined in DataSourceOptions.scala for OPERATION_OPT_KEY but care must be taken to ensure we do not cause backwards compatibility issue by changing the property value. In other words, we need to retain the lower case values ("bulk_insert", "insert" and "upsert") but make it an enum.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)