You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/28 00:57:46 UTC
[GitHub] [spark] huaxingao opened a new pull request #29885: Make DataFrameWriter.jdbc work for DataSource V2
huaxingao opened a new pull request #29885:
URL: https://github.com/apache/spark/pull/29885
### What changes were proposed in this pull request?
Support multiple catalogs in the following use case:
```
DataFrameWriter.jdbc(url, "catalog.db.tbl", properties)
```
### Why are the changes needed?
Make DataFrameWriter.jdbc work for DataSource V2
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
New test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704603144
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34078/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702957113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703147385
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33986/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702544796
cc @dongjoon-hyun @maropu @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702861891
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702932157
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703138333
**[Test build #129378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129378/testReport)** for PR 29885 at commit [`0a7c845`](https://github.com/apache/spark/commit/0a7c84559e89d58fd7bc41c73eaad9500379cd34).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704603157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704595417
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34078/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702872829
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704578594
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao closed pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao closed pull request #29885:
URL: https://github.com/apache/spark/pull/29885
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703011638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699776010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699775426
**[Test build #129160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129160/testReport)** for PR 29885 at commit [`c7ea80f`](https://github.com/apache/spark/commit/c7ea80fba5db62db3b9522f56725ebe2def12043).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703168791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703138333
**[Test build #129378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129378/testReport)** for PR 29885 at commit [`0a7c845`](https://github.com/apache/spark/commit/0a7c84559e89d58fd7bc41c73eaad9500379cd34).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704603157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703168597
**[Test build #129378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129378/testReport)** for PR 29885 at commit [`0a7c845`](https://github.com/apache/spark/commit/0a7c84559e89d58fd7bc41c73eaad9500379cd34).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703011638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702933292
**[Test build #129365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129365/testReport)** for PR 29885 at commit [`19441da`](https://github.com/apache/spark/commit/19441da91073a48aa07e5af6642cb1cea667861e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699776010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703147389
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704453447
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703168791
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703147389
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704472076
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34065/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699716840
**[Test build #129160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129160/testReport)** for PR 29885 at commit [`c7ea80f`](https://github.com/apache/spark/commit/c7ea80fba5db62db3b9522f56725ebe2def12043).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703144659
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33986/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29885:
URL: https://github.com/apache/spark/pull/29885#discussion_r500652507
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,24 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
- this.extraOptions ++= Seq("url" -> url, "dbtable" -> table)
- format("jdbc").save()
+ // explicit url should override all
+ this.extraOptions += "url" -> url
+
+ import df.sparkSession.sessionState.analyzer.NonSessionCatalogAndIdentifier
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+
+ val session = df.sparkSession
+ this.source = "jdbc"
+
+ session.sessionState.sqlParser.parseMultipartIdentifier(table) match {
+ case nameParts @ NonSessionCatalogAndIdentifier(catalog, tableIdentifier) =>
+ saveAsTable(catalog.asTableCatalog, tableIdentifier, nameParts)
Review comment:
Returned catalog is always a `TableCatalog`?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,24 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
- this.extraOptions ++= Seq("url" -> url, "dbtable" -> table)
- format("jdbc").save()
+ // explicit url should override all
+ this.extraOptions += "url" -> url
+
+ import df.sparkSession.sessionState.analyzer.NonSessionCatalogAndIdentifier
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+
+ val session = df.sparkSession
+ this.source = "jdbc"
+
+ session.sessionState.sqlParser.parseMultipartIdentifier(table) match {
+ case nameParts @ NonSessionCatalogAndIdentifier(catalog, tableIdentifier) =>
+ saveAsTable(catalog.asTableCatalog, tableIdentifier, nameParts)
Review comment:
Or do we need to check if `catalog` is `JDBCTableCatalog`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702861891
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704472030
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34065/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704472061
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699730134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29885:
URL: https://github.com/apache/spark/pull/29885#discussion_r501185283
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,24 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
- this.extraOptions ++= Seq("url" -> url, "dbtable" -> table)
- format("jdbc").save()
+ // explicit url should override all
+ this.extraOptions += "url" -> url
+
+ import df.sparkSession.sessionState.analyzer.NonSessionCatalogAndIdentifier
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+
+ val session = df.sparkSession
+ this.source = "jdbc"
+
+ session.sessionState.sqlParser.parseMultipartIdentifier(table) match {
+ case nameParts @ NonSessionCatalogAndIdentifier(catalog, tableIdentifier) =>
+ saveAsTable(catalog.asTableCatalog, tableIdentifier, nameParts)
Review comment:
Seems to me that the catalog in `NonSessionCatalogAndIdentifier` is always a `TableCatalog`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704666432
**[Test build #129471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129471/testReport)** for PR 29885 at commit [`9ff4313`](https://github.com/apache/spark/commit/9ff43135256f7c3ad3922f6772186634d69633a1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703854396
@MaxGekk Could you please review this one too? Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704581400
**[Test build #129471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129471/testReport)** for PR 29885 at commit [`9ff4313`](https://github.com/apache/spark/commit/9ff43135256f7c3ad3922f6772186634d69633a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-705294163
cc @cloud-fan I think the holidays are not over yet, but just in case you have some time to work :) Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702957113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29885:
URL: https://github.com/apache/spark/pull/29885#discussion_r498867658
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,26 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
- this.extraOptions ++= Seq("url" -> url, "dbtable" -> table)
- format("jdbc").save()
+ this.extraOptions ++= Seq("url" -> url)
+
+ import df.sparkSession.sessionState.analyzer.{AsTableIdentifier, NonSessionCatalogAndIdentifier}
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+
+ val session = df.sparkSession
+ format("jdbc")
Review comment:
Why did you call `format` here? `this.source = "jdbc"` insead?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,26 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
- this.extraOptions ++= Seq("url" -> url, "dbtable" -> table)
- format("jdbc").save()
+ this.extraOptions ++= Seq("url" -> url)
+
+ import df.sparkSession.sessionState.analyzer.{AsTableIdentifier, NonSessionCatalogAndIdentifier}
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+
+ val session = df.sparkSession
+ format("jdbc")
+
+ session.sessionState.sqlParser.parseMultipartIdentifier(table) match {
+ case nameParts @ NonSessionCatalogAndIdentifier(catalog, tableIdentifier) =>
+ saveAsTable(catalog.asTableCatalog, tableIdentifier, nameParts)
+
+ case AsTableIdentifier(_) =>
+ this.extraOptions ++= Seq("dbtable" -> table)
+ saveToV1Source(None)
+
+ case other =>
+ throw new AnalysisException(
+ s"Couldn't find a catalog to handle the identifier ${other.quoted}.")
Review comment:
We need this error handling? Rather, this is it like this?
```
session.sessionState.sqlParser.parseMultipartIdentifier(table) match {
case nameParts @ NonSessionCatalogAndIdentifier(catalog, tableIdentifier) =>
saveAsTable(catalog.asTableCatalog, tableIdentifier, nameParts)
case _ =>
// explicit dbtable should override all
this.extraOptions ++= Seq("dbtable" -> table)
saveToV1Source(None)
}
```
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala
##########
@@ -221,4 +221,36 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession {
checkAnswer(sql("SELECT name, id FROM h2.test.abc"), Row("bob", 4))
}
}
+
+ test("dfwriter.jdbc") {
Review comment:
Could you make this test title clearer?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -808,9 +808,26 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
assertNotBucketed("jdbc")
// connectionProperties should override settings in extraOptions.
this.extraOptions ++= connectionProperties.asScala
- // explicit url and dbtable should override all
Review comment:
Could you keep this comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704581400
**[Test build #129471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129471/testReport)** for PR 29885 at commit [`9ff4313`](https://github.com/apache/spark/commit/9ff43135256f7c3ad3922f6772186634d69633a1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702828547
**[Test build #129356 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129356/testReport)** for PR 29885 at commit [`19441da`](https://github.com/apache/spark/commit/19441da91073a48aa07e5af6642cb1cea667861e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-703011273
**[Test build #129365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129365/testReport)** for PR 29885 at commit [`19441da`](https://github.com/apache/spark/commit/19441da91073a48aa07e5af6642cb1cea667861e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699716840
**[Test build #129160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129160/testReport)** for PR 29885 at commit [`c7ea80f`](https://github.com/apache/spark/commit/c7ea80fba5db62db3b9522f56725ebe2def12043).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702872829
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704578594
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704666955
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702872840
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129356/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704472061
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704430093
**[Test build #129458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129458/testReport)** for PR 29885 at commit [`0b00020`](https://github.com/apache/spark/commit/0b00020b3518d3b3601571c4401f5bc95814287e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao closed pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao closed pull request #29885:
URL: https://github.com/apache/spark/pull/29885
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704577870
**[Test build #129458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129458/testReport)** for PR 29885 at commit [`0b00020`](https://github.com/apache/spark/commit/0b00020b3518d3b3601571c4401f5bc95814287e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702850143
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33966/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702861364
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33966/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704430093
**[Test build #129458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129458/testReport)** for PR 29885 at commit [`0b00020`](https://github.com/apache/spark/commit/0b00020b3518d3b3601571c4401f5bc95814287e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702957091
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33975/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #29885:
URL: https://github.com/apache/spark/pull/29885#discussion_r500542489
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala
##########
@@ -221,4 +221,36 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession {
checkAnswer(sql("SELECT name, id FROM h2.test.abc"), Row("bob", 4))
}
}
+
+ test("DataFrameWriter: jdbc") {
Review comment:
Other tests above have in their title `DataFrameWriterV2` but this one `DataFrameWriter`. Do you suppose to test v1 mostly here?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala
##########
@@ -221,4 +221,36 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession {
checkAnswer(sql("SELECT name, id FROM h2.test.abc"), Row("bob", 4))
}
}
+
+ test("DataFrameWriter: jdbc") {
+ withTable("h2.test.abc") {
+ sql("CREATE TABLE h2.test.abc USING _ AS SELECT * FROM h2.test.people")
+ val df1 = Seq(("evan", 3)).toDF("NAME", "ID")
+ val df2 = Seq(("cathy", 4)).toDF("NAME", "ID")
+ val properties = new Properties()
+
+ df1.write
+ .mode("append")
+ .jdbc(url, "h2.test.abc", properties)
+ checkAnswer(sql("SELECT name, id FROM h2.test.abc"),
+ Seq(Row("fred", 1), Row("mary", 2), Row("evan", 3)))
Review comment:
```suggestion
checkAnswer(
sql("SELECT name, id FROM h2.test.abc"),
Seq(Row("fred", 1), Row("mary", 2), Row("evan", 3)))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #29885:
URL: https://github.com/apache/spark/pull/29885#discussion_r500625623
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCV2Suite.scala
##########
@@ -221,4 +221,36 @@ class JDBCV2Suite extends QueryTest with SharedSparkSession {
checkAnswer(sql("SELECT name, id FROM h2.test.abc"), Row("bob", 4))
}
}
+
+ test("DataFrameWriter: jdbc") {
Review comment:
Other tests have titles `DataFrameWriterV2` because these are testing `DataFrameWriterV2` APIs such as `DataFrameWriterV2.create` or `DataFrameWriterV2.replace`. `jdbc` is a `DataFrameWriter` API.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702950005
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33975/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699730134
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699730126
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33775/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-699725295
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/33775/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702828230
@maropu Thanks for reviewing. Code updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-702933292
**[Test build #129365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129365/testReport)** for PR 29885 at commit [`19441da`](https://github.com/apache/spark/commit/19441da91073a48aa07e5af6642cb1cea667861e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29885: [SPARK-33010][SQL]Make DataFrameWriter.jdbc work for DataSource V2
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29885:
URL: https://github.com/apache/spark/pull/29885#issuecomment-704666955
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org