You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/30 08:04:24 UTC
[GitHub] [spark] huaxingao opened a new pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
huaxingao opened a new pull request #34754:
URL: https://github.com/apache/spark/pull/34754
### What changes were proposed in this pull request?
This PR migrates `ReplaceTableAsSelectStatement` to the v2 command
### Why are the changes needed?
Migrate to the standard V2 framework
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982394191
**[Test build #145757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145757/testReport)** for PR 34754 at commit [`3323d2e`](https://github.com/apache/spark/commit/3323d2e1312178cee39368012a7a06d0704fee23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983814722
**[Test build #145820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145820/testReport)** for PR 34754 at commit [`ecc596c`](https://github.com/apache/spark/commit/ecc596ccfd38fd9920504ed8eb17e0eb81969397).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983814722
**[Test build #145820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145820/testReport)** for PR 34754 at commit [`ecc596c`](https://github.com/apache/spark/commit/ecc596ccfd38fd9920504ed8eb17e0eb81969397).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r760421099
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,22 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
+ val tableSpec = TableSpec(
+ bucketSpec = None,
+ properties = Map.empty,
+ provider = Some(source),
+ options = Map.empty,
+ location = extraOptions.get("path"),
+ comment = extraOptions.get(TableCatalog.PROP_COMMENT),
+ serde = None,
+ external = false)
+ ReplaceTableAsSelect(
+ UnresolvedDBObjectName(nameParts, isNamespace = false),
Review comment:
shall we prepend `catalog.name`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982631688
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145757/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982430877
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50229/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982495100
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50229/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983862023
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50295/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982495100
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50229/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-984159907
Merged to master. Thank you very much for reviewing! @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-984077421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145820/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r771152985
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,22 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
+ val tableSpec = TableSpec(
+ bucketSpec = None,
+ properties = Map.empty,
+ provider = Some(source),
+ options = Map.empty,
+ location = extraOptions.get("path"),
+ comment = extraOptions.get(TableCatalog.PROP_COMMENT),
+ serde = None,
+ external = false)
+ ReplaceTableAsSelect(
+ UnresolvedDBObjectName(nameParts, isNamespace = false),
partitioningAsV2,
- None,
- Map.empty,
- Some(source),
- Map.empty,
- extraOptions.get("path"),
- extraOptions.get(TableCatalog.PROP_COMMENT),
- extraOptions.toMap,
- None,
- orCreate = true) // Create the table if it doesn't exist
+ df.queryExecution.analyzed,
+ tableSpec,
+ writeOptions = Map.empty,
Review comment:
Sorry for the bug. Will fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r760456645
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,22 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
+ val tableSpec = TableSpec(
+ bucketSpec = None,
+ properties = Map.empty,
+ provider = Some(source),
+ options = Map.empty,
+ location = extraOptions.get("path"),
+ comment = extraOptions.get(TableCatalog.PROP_COMMENT),
+ serde = None,
+ external = false)
+ ReplaceTableAsSelect(
+ UnresolvedDBObjectName(nameParts, isNamespace = false),
Review comment:
Seems no need? I stepped into one of the test, the `nameParts` here is "testcat", "ns", "t", so it already contains `catalog.name`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r771059560
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,22 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
+ val tableSpec = TableSpec(
+ bucketSpec = None,
+ properties = Map.empty,
+ provider = Some(source),
+ options = Map.empty,
+ location = extraOptions.get("path"),
+ comment = extraOptions.get(TableCatalog.PROP_COMMENT),
+ serde = None,
+ external = false)
+ ReplaceTableAsSelect(
+ UnresolvedDBObjectName(nameParts, isNamespace = false),
partitioningAsV2,
- None,
- Map.empty,
- Some(source),
- Map.empty,
- extraOptions.get("path"),
- extraOptions.get(TableCatalog.PROP_COMMENT),
- extraOptions.toMap,
- None,
- orCreate = true) // Create the table if it doesn't exist
+ df.queryExecution.analyzed,
+ tableSpec,
+ writeOptions = Map.empty,
Review comment:
I think there is a bug here. Previously we pass `extraOptions.toMap` as the write options, now we don't. @huaxingao can you help to fix it with a test case? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983102159
cc @cloud-fan Could you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-984075965
**[Test build #145820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145820/testReport)** for PR 34754 at commit [`ecc596c`](https://github.com/apache/spark/commit/ecc596ccfd38fd9920504ed8eb17e0eb81969397).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r759937291
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##########
@@ -196,11 +204,19 @@ case class AtomicReplaceTableAsSelectExec(
partitioning: Seq[Transform],
plan: LogicalPlan,
query: SparkPlan,
- properties: Map[String, String],
+ tableSpec: TableSpec,
writeOptions: CaseInsensitiveStringMap,
orCreate: Boolean,
invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends TableWriteExecHelper {
+ val properties = {
+ val props = CatalogV2Util.convertTableProperties(
Review comment:
nit: should the util function take `TableSpec` directly? and also fill the ownership property. Then we can simplify the code here and other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-984077421
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145820/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r760355475
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala
##########
@@ -196,11 +204,19 @@ case class AtomicReplaceTableAsSelectExec(
partitioning: Seq[Transform],
plan: LogicalPlan,
query: SparkPlan,
- properties: Map[String, String],
+ tableSpec: TableSpec,
writeOptions: CaseInsensitiveStringMap,
orCreate: Boolean,
invalidateCache: (TableCatalog, Table, Identifier) => Unit) extends TableWriteExecHelper {
+ val properties = {
+ val props = CatalogV2Util.convertTableProperties(
Review comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983936753
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50295/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982631688
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145757/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r759935005
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,13 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
- partitioningAsV2,
- None,
- Map.empty,
- Some(source),
- Map.empty,
- extraOptions.get("path"),
- extraOptions.get(TableCatalog.PROP_COMMENT),
- extraOptions.toMap,
- None,
- orCreate = true) // Create the table if it doesn't exist
+ val tableSpec = TableSpec(None, Map.empty, Some(source), Map.empty,
Review comment:
let's use named parameter here: `TableSpec(bucketSpec = None, ...)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao closed pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao closed pull request #34754:
URL: https://github.com/apache/spark/pull/34754
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983938196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50295/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982466932
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50229/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982627038
**[Test build #145757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145757/testReport)** for PR 34754 at commit [`3323d2e`](https://github.com/apache/spark/commit/3323d2e1312178cee39368012a7a06d0704fee23).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-982394191
**[Test build #145757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145757/testReport)** for PR 34754 at commit [`3323d2e`](https://github.com/apache/spark/commit/3323d2e1312178cee39368012a7a06d0704fee23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r759935336
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala
##########
@@ -195,20 +195,12 @@ final class DataFrameWriterV2[T] private[sql](table: String, ds: Dataset[T])
}
private def internalReplace(orCreate: Boolean): Unit = {
- runCommand(
- ReplaceTableAsSelectStatement(
- tableName,
- logicalPlan,
- partitioning.getOrElse(Seq.empty),
- None,
- properties.toMap,
- provider,
- Map.empty,
- None,
- None,
- options.toMap,
- None,
- orCreate = orCreate))
+ val tableSpec = TableSpec(None, properties.toMap, provider, Map.empty,
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34754:
URL: https://github.com/apache/spark/pull/34754#discussion_r760355203
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
##########
@@ -586,19 +586,13 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) {
AppendData.byName(v2Relation, df.logicalPlan, extraOptions.toMap)
case (SaveMode.Overwrite, _) =>
- ReplaceTableAsSelectStatement(
- nameParts,
- df.queryExecution.analyzed,
- partitioningAsV2,
- None,
- Map.empty,
- Some(source),
- Map.empty,
- extraOptions.get("path"),
- extraOptions.get(TableCatalog.PROP_COMMENT),
- extraOptions.toMap,
- None,
- orCreate = true) // Create the table if it doesn't exist
+ val tableSpec = TableSpec(None, Map.empty, Some(source), Map.empty,
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34754: [SPARK-37496][SQL] Migrate ReplaceTableAsSelectStatement to v2 command
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34754:
URL: https://github.com/apache/spark/pull/34754#issuecomment-983938196
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50295/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org