[GitHub] cloud-fan opened a new pull request #23859: [SPARK-26956][SQL] translate streaming output mode to write operators

[GitBox <gi...@apache.org>]

cloud-fan opened a new pull request #23859: [SPARK-26956][SQL] translate streaming output mode to write operators
URL: https://github.com/apache/spark/pull/23859
 
 
   ## What changes were proposed in this pull request?
   
   Similar to `SaveMode`, we should remove streaming `OutputMode` from data source v2 API, and use operations that has clear semantic.
   
   The changes are:
   1. append mode: create `StreamingWrite` directly. By default, the `WriteBuilder` will create `Write` to append data.
   2. complete mode: call `SupportsTruncate#truncate`. Complete mode means truncating all the old data and appending new data, and `SupportsTruncate` has exactly the same semantic.
   3. update mode: fail. The current streaming framework can't propagate the update keys, so v2 sinks are not able to implement update mode. In the future we can introduce a `SupportsUpdate` trait.
   
   The behavior changes:
   1. all the v2 sinks(foreach, console, memory, kafka, noop) don't support update mode. The fact is, previously all the v2 sinks implement the update mode wrong. None of them can really support it.
   2. kafka sink doesn't support complete mode. The fact is, the kafka sink can only append data.
   
   ## How was this patch tested?
   
   existing tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org