[GitHub] [spark] rdblue opened a new pull request #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API

[GitBox <gi...@apache.org>]

rdblue opened a new pull request #25354: [SPARK-28612][SQL] Add DataFrameWriterV2 API
URL: https://github.com/apache/spark/pull/25354
 
 
   ## What changes were proposed in this pull request?
   
   This adds a new write API as proposed in the [SPIP to standardize logical plans](https://issues.apache.org/jira/browse/SPARK-23521). This new API:
   
   * Uses clear verbs to execute writes, like `append`, `overwrite`, `create`, and `replace` that correspond to the new logical plans.
   * Only creates v2 logical plans so the behavior is always consistent.
   * Does not allow table configuration options for operations that cannot change table configuration. For example, `partitionedBy` can only be called when the writer executes `create` or `replace`.
   
   Here are a few example uses of the new API:
   
   ```scala
   df.writeTo("catalog.db.table").append()
   df.writeTo("catalog.db.table").overwrite($"date" === "2019-06-01")
   df.writeTo("catalog.db.table").overwritePartitions()
   df.writeTo("catalog.db.table").asParquet.create()
   df.writeTo("catalog.db.table").partitionedBy(days($"ts")).createOrReplace()
   df.writeTo("catalog.db.table").using("abc").replace()
   ```
   
   ## How was this patch tested?
   
   Added `DataFrameWriterV2Suite` that tests the new write API. Existing tests for v2 plans.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org