You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ueshin (via GitHub)" <gi...@apache.org> on 2023/02/13 23:10:06 UTC
[GitHub] [spark] ueshin opened a new pull request, #40000: [SPARK-41818][SPARK-42000][CONNECT] Fix saveAsTable to find the default source
ueshin opened a new pull request, #40000:
URL: https://github.com/apache/spark/pull/40000
### What changes were proposed in this pull request?
Fixes `DataFrameWriter.saveAsTable` to find the default source.
### Why are the changes needed?
Currently `DataFrameWriter.saveAsTable` fails when `format` is not specified because protobuf defines `source` as required and it will be an empty string instead of `null`, then `DataFrameWriter` tries to find the data source `""`.
The `source` field should be optional to let Spark decide the default source.
### Does this PR introduce _any_ user-facing change?
Users can call `saveAsTable` without `format`.
### How was this patch tested?
Enabled related tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on a diff in pull request #40000: [SPARK-41818][SPARK-42000][CONNECT] Fix saveAsTable to find the default source
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #40000:
URL: https://github.com/apache/spark/pull/40000#discussion_r1105114706
##########
python/pyspark/sql/readwriter.py:
##########
@@ -468,7 +468,7 @@ def table(self, tableName: str) -> "DataFrame":
| 8|
| 9|
+---+
- >>> _ = spark.sql("DROP TABLE tblA")
+ >>> _ = spark.sql("DROP TABLE tblA").collect()
Review Comment:
For [SPARK-41725](https://issues.apache.org/jira/browse/SPARK-41725).
##########
python/pyspark/sql/connect/readwriter.py:
##########
@@ -472,6 +472,8 @@ def save(
def insertInto(self, tableName: str, overwrite: Optional[bool] = None) -> None:
if overwrite is not None:
self.mode("overwrite" if overwrite else "append")
+ elif self._write.mode is None or self._write.mode != "overwrite":
+ self.mode("append")
self.saveAsTable(tableName)
Review Comment:
I guess we shouldn't reuse this for `insertInto`? Should revisit this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #40000: [SPARK-41818][SPARK-42000][CONNECT] Fix saveAsTable to find the default source
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #40000:
URL: https://github.com/apache/spark/pull/40000#issuecomment-1428890417
Merged to master and branch-3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #40000: [SPARK-41818][SPARK-42000][CONNECT] Fix saveAsTable to find the default source
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #40000: [SPARK-41818][SPARK-42000][CONNECT] Fix saveAsTable to find the default source
URL: https://github.com/apache/spark/pull/40000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org