You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/21 16:52:10 UTC
[GitHub] [spark] huaxingao opened a new pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
huaxingao opened a new pull request #34060:
URL: https://github.com/apache/spark/pull/34060
### What changes were proposed in this pull request?
Migrate CreateTableStatement to v2 command framework
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924267078
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971895002
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49817/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971859406
**[Test build #145346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145346/testReport)** for PR 34060 at commit [`6ffae91`](https://github.com/apache/spark/commit/6ffae91ee87d74fbd11c4cc62c9282f85a724702).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r752797321
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,21 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
+ val cmd = CreateV2Table(
+ UnresolvedDBObjectName(
+ originalMultipartIdentifier,
+ isNamespace = true),
Review comment:
why is it `true`? i guess it is a `table` name
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,21 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
+ val cmd = CreateV2Table(
+ UnresolvedDBObjectName(
+ originalMultipartIdentifier,
+ isNamespace = true),
df.schema.asNullable,
partitioningColumns.getOrElse(Nil).asTransforms.toSeq,
None,
Map.empty[String, String],
- Some(source),
Map.empty[String, String],
- extraOptions.get("path"),
None,
+ extraOptions.get("path"),
None,
+ Some(source),
external = false,
- ifNotExists = false)
+ ignoreIfExists = false)
Review comment:
Just for my information, is `ignoreIfExists` like `ifNotExists` of `CreateTableStatement`? Thanks!
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##########
@@ -306,11 +306,6 @@ private[sql] object CatalogV2Util {
catalog.name().equalsIgnoreCase(CatalogManager.SESSION_CATALOG_NAME)
}
- def convertTableProperties(c: CreateTableStatement): Map[String, String] = {
Review comment:
Can we just change `c` to `CreateV2Table`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982246781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978409113
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50063/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r756896465
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala
##########
@@ -32,10 +33,16 @@ case class CreateTableExec(
identifier: Identifier,
tableSchema: StructType,
partitioning: Seq[Transform],
- tableProperties: Map[String, String],
+ tableSpec: TableSpec,
ignoreIfExists: Boolean) extends LeafV2CommandExec {
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+ val props = CatalogV2Util.convertTableProperties(
Review comment:
let's not expose this as it's intermediate
```
val tableProperties = {
val props = ...
CatalogV2Util.withDefaultOwnership(props)
}
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableExec.scala
##########
@@ -32,10 +33,16 @@ case class CreateTableExec(
identifier: Identifier,
tableSchema: StructType,
partitioning: Seq[Transform],
- tableProperties: Map[String, String],
+ tableSpec: TableSpec,
ignoreIfExists: Boolean) extends LeafV2CommandExec {
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+ val props = CatalogV2Util.convertTableProperties(
Review comment:
let's not expose this as it's intermediate. how about
```
val tableProperties = {
val props = ...
CatalogV2Util.withDefaultOwnership(props)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975333196
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49977/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755773462
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##########
@@ -263,6 +265,51 @@ case class PreprocessTableCreation(sparkSession: SparkSession) extends Rule[Logi
create.withPartitioning(normalizedPartitions)
}
+
+ case create: V2CreateTablePlanX if create.childrenResolved =>
+ val schema = create.tableSchema
+ val partitioning = create.partitioning
+ val name = create.name.asInstanceOf[ResolvedDBObjectName].nameParts
+ val identifier = if (name.length == 2) {
+ Identifier.of(Array(name(0)), name(1))
+ } else {
+ Identifier.of(Array.empty, name(0))
Review comment:
what if the `name` has more than 2 parts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975395328
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49977/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
dchvn commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-969851943
Hi @huaxingao , any update? I am trying to migrate `ReplaceTableStatement` by learn from your PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928832541
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-936792421
**[Test build #143889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143889/testReport)** for PR 34060 at commit [`0af17cd`](https://github.com/apache/spark/commit/0af17cdd706d00eccad9e12c984b24dfcd650e05).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977673271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755772048
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##########
@@ -25,9 +25,11 @@ import org.apache.spark.sql.catalyst.catalog._
import org.apache.spark.sql.catalyst.expressions.{Expression, InputFileBlockLength, InputFileBlockStart, InputFileName, RowOrdering}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.Identifier
import org.apache.spark.sql.connector.expressions.{FieldReference, RewritableTransform}
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.command.DDLUtils
+import org.apache.spark.sql.execution.datasources.{CreateTable => DataSourceCreateTable}
Review comment:
```suggestion
import org.apache.spark.sql.execution.datasources.{CreateTable => CreateTableV1}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755784540
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
##########
@@ -156,9 +156,16 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
}
WriteToDataSourceV2Exec(writer, invalidateCacheFunc, planLater(query), customMetrics) :: Nil
- case CreateV2Table(catalog, ident, schema, parts, props, ifNotExists) =>
+ case CreateTable(ResolvedDBObjectName(catalog, ident), schema, partitioning, bucketSpec,
+ tableSpec, ifNotExists) =>
+ val props = CatalogV2Util.convertTableProperties(
Review comment:
I think it's better to keep the strategy rule simple and move the logic of generating table properties into `CreateTableExec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977622901
**[Test build #145569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145569/testReport)** for PR 34060 at commit [`6a3bc84`](https://github.com/apache/spark/commit/6a3bc845e618b10b10e52bfc6d575beb84d5d64d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977673271
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145568/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r766005755
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,23 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
- df.schema.asNullable,
- partitioningColumns.getOrElse(Nil).asTransforms.toSeq,
+ val tableProperties = TableSpec(
Review comment:
Will have a follow up to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982156175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-936868281
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755708588
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
##########
@@ -143,25 +143,25 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
// For CREATE TABLE [AS SELECT], we should use the v1 command if the catalog is resolved to the
// session catalog and the table provider is not v2.
- case c @ CreateTableStatement(
- SessionCatalogAndTable(catalog, tbl), _, _, _, _, _, _, _, _, _, _, _) =>
+ case c @ CreateV2Table(ResolvedDBObjectName(catalog, name), _, _, _, _, _) =>
Review comment:
I think we should only change the plan if the catalog is session catalog. The `CreateTableExec` should be responsible to convert `TableSpec` to table properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755707836
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -1090,3 +1106,12 @@ case class DropIndex(
override protected def withNewChildInternal(newChild: LogicalPlan): DropIndex =
copy(table = newChild)
}
+
+case class TableProperties(
Review comment:
This name is a bit misleading. How about `TableSpec`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979844619
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977667786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978409113
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50063/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755786855
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -189,17 +190,35 @@ trait V2CreateTablePlan extends LogicalPlan {
def withPartitioning(rewritten: Seq[Transform]): V2CreateTablePlan
}
+trait V2CreateTablePlanX extends LogicalPlan {
Review comment:
Do we have to add this? Can we let the new `CreateTable` extends `V2CreateTablePlan` and implement `tableName`?
```
def tableName = {
assert(name.isResolved)
name.asInstanceOf[ResolvedDBObjectName].nameParts.asIdentifier
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977710744
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971971611
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49817/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982147743
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928832541
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928702662
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928702662
**[Test build #143667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143667/testReport)** for PR 34060 at commit [`9e42d67`](https://github.com/apache/spark/commit/9e42d67de52fa000bed3f21e40527360934a56f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982106173
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982243298
**[Test build #145733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145733/testReport)** for PR 34060 at commit [`8fdf059`](https://github.com/apache/spark/commit/8fdf059093623b8d36e7231ac539dbb4a6449f40).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977661304
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979797257
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977587562
**[Test build #145568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145568/testReport)** for PR 34060 at commit [`29ba020`](https://github.com/apache/spark/commit/29ba020e2d3d8da02bb433d85b47a4d117dcce78).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755707733
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3469,9 +3469,14 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
case _ =>
// Note: table schema includes both the table columns list and the partition columns
// with data type.
+ val tableProperties = TableProperties(properties, provider, options, location, comment,
+ serdeInfo, external)
val schema = StructType(columns ++ partCols)
- CreateTableStatement(table, schema, partitioning, bucketSpec, properties, provider,
- options, location, comment, serdeInfo, external = external, ifNotExists = ifNotExists)
+ CreateV2Table(
Review comment:
The naming is a bit hard here. The best name should be `CreateTable`, but it's already taken. `CreateV2Table` is not proper here as this can also create v1 tables.
Given that we can will migrate all the commands eventually, the existing `CreateTable` will be removed at the end. How about we still use the name `CreateTable` here? i.e. add a new `CreateTable` logical plan in `org.apache.spark.sql.catalyst.plans.logical`, which does not conflict with the existing `CreateTable`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
dchvn commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r752799103
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,21 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
+ val cmd = CreateV2Table(
+ UnresolvedDBObjectName(
+ originalMultipartIdentifier,
+ isNamespace = true),
df.schema.asNullable,
partitioningColumns.getOrElse(Nil).asTransforms.toSeq,
None,
Map.empty[String, String],
- Some(source),
Map.empty[String, String],
- extraOptions.get("path"),
None,
+ extraOptions.get("path"),
None,
+ Some(source),
external = false,
- ifNotExists = false)
+ ignoreIfExists = false)
Review comment:
Just for my information, does `ignoreIfExists` like `ifNotExists` of `CreateTableStatement`? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975349262
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r713566120
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##########
@@ -263,6 +264,51 @@ case class PreprocessTableCreation(sparkSession: SparkSession) extends Rule[Logi
create.withPartitioning(normalizedPartitions)
}
+
+ case create: V2CreateTablePlanX if create.childrenResolved =>
Review comment:
The original `case create: V2CreateTablePlan` will be removed and this will be changed to
`case create: V2CreateTablePlan` after the migration of `CreateTableAsSelect`, `ReplaceTable` and `ReplaceTableAsSelect` is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928757002
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r756898767
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##########
@@ -481,19 +481,26 @@ class PlanResolutionSuite extends AnalysisTest {
"other" -> "20")
parseAndResolve(sql) match {
- case create: CreateV2Table =>
- assert(create.catalog.name == "testcat")
- assert(create.tableName == Identifier.of(Array("mydb"), "table_name"))
+ case create: CatalystCreateTable =>
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].catalog.name == "testcat")
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].nameParts.mkString(".") ==
+ "mydb.table_name")
assert(create.tableSchema == new StructType()
.add("id", LongType)
.add("description", StringType)
.add("point", new StructType().add("x", DoubleType).add("y", DoubleType)))
assert(create.partitioning.isEmpty)
- assert(create.properties == expectedProperties)
Review comment:
do we have end-to-end test for CREATE TABLE? we can test it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r756891808
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3469,9 +3469,14 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
case _ =>
// Note: table schema includes both the table columns list and the partition columns
// with data type.
+ val tableSpec = TableSpec(bucketSpec, properties, provider, options, location, comment,
+ serdeInfo, external)
val schema = StructType(columns ++ partCols)
- CreateTableStatement(table, schema, partitioning, bucketSpec, properties, provider,
- options, location, comment, serdeInfo, external = external, ifNotExists = ifNotExists)
+ CreateTable(
+ UnresolvedDBObjectName(
+ table,
+ isNamespace = false),
Review comment:
```suggestion
UnresolvedDBObjectName(table, isNamespace = false),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979771136
**[Test build #145661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145661/testReport)** for PR 34060 at commit [`39320f4`](https://github.com/apache/spark/commit/39320f4f12546475cea54a9914d05973849bb89b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979966123
**[Test build #145661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145661/testReport)** for PR 34060 at commit [`39320f4`](https://github.com/apache/spark/commit/39320f4f12546475cea54a9914d05973849bb89b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979967448
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977667786
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-974926472
@dchvn I need to work with @cloud-fan on this PR. Once this PR is done, you can continue your Migrate CreateTableAsSelectStatement PR. I will be extreme slow because I am taking off until 11/29.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975216033
**[Test build #145505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145505/testReport)** for PR 34060 at commit [`a2dd853`](https://github.com/apache/spark/commit/a2dd853d3ab8d17cc26d50b183d8363f00ad8bbc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r713565953
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -189,17 +190,35 @@ trait V2CreateTablePlan extends LogicalPlan {
def withPartitioning(rewritten: Seq[Transform]): V2CreateTablePlan
}
+trait V2CreateTablePlanX extends LogicalPlan {
Review comment:
`V2CreateTablePlan` can't be deleted yet because it is also extended by `CreateTableAsSelect`, `ReplaceTable` and `ReplaceTableAsSelect`. I use `V2CreateTablePlanX` for this new trait for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924223735
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977622901
**[Test build #145569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145569/testReport)** for PR 34060 at commit [`6a3bc84`](https://github.com/apache/spark/commit/6a3bc845e618b10b10e52bfc6d575beb84d5d64d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975349262
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145505/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r756892666
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -3402,7 +3402,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg
}
/**
- * Create a table, returning a [[CreateTableStatement]] logical plan.
+ * Create a table, returning a [[CreateTable]] or [[CreateV2Table]] logical plan.
Review comment:
```suggestion
* Create a table, returning a [[CreateTable]] or [[CreateTableAsSelectStatement]] logical plan.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978345564
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50063/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r758283132
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -819,6 +820,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
redactMapString(map.asCaseSensitiveMap().asScala, maxFields)
case map: Map[_, _] =>
redactMapString(map, maxFields)
+ case t: TableSpec =>
+ redactMapString(t.options, maxFields)
+ redactMapString(t.properties, maxFields)
Review comment:
I think we need to return a copy of `TableSpec` with options and properties redacted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r758519908
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##########
@@ -481,19 +481,26 @@ class PlanResolutionSuite extends AnalysisTest {
"other" -> "20")
parseAndResolve(sql) match {
- case create: CreateV2Table =>
- assert(create.catalog.name == "testcat")
- assert(create.tableName == Identifier.of(Array("mydb"), "table_name"))
+ case create: CatalystCreateTable =>
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].catalog.name == "testcat")
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].nameParts.mkString(".") ==
+ "mydb.table_name")
assert(create.tableSchema == new StructType()
.add("id", LongType)
.add("description", StringType)
.add("point", new StructType().add("x", DoubleType).add("y", DoubleType)))
assert(create.partitioning.isEmpty)
- assert(create.properties == expectedProperties)
Review comment:
We have an end to end test here for table properties https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala#L123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982348104
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979967448
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145661/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977721539
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145569/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dchvn edited a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
dchvn edited a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-969851943
Hi @huaxingao , any update? I am trying to migrate `ReplaceTableStatement` and `CreateTableAsSelectStatement` by learn from your PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971824875
@dchvn Sorry for the late reply. I will work on this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924180831
**[Test build #143477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143477/testReport)** for PR 34060 at commit [`1965f77`](https://github.com/apache/spark/commit/1965f773825e4f6cd14b71d7f3d32550daaad4c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928832541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928822193
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48179/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928963268
**[Test build #143667 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143667/testReport)** for PR 34060 at commit [`9e42d67`](https://github.com/apache/spark/commit/9e42d67de52fa000bed3f21e40527360934a56f7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-936998047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-936998047
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971859406
**[Test build #145346 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145346/testReport)** for PR 34060 at commit [`6ffae91`](https://github.com/apache/spark/commit/6ffae91ee87d74fbd11c4cc62c9282f85a724702).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977642156
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977721207
**[Test build #145569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145569/testReport)** for PR 34060 at commit [`6a3bc84`](https://github.com/apache/spark/commit/6a3bc845e618b10b10e52bfc6d575beb84d5d64d).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975372684
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49977/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755769160
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##########
@@ -190,17 +191,32 @@ trait V2CreateTablePlan extends LogicalPlan {
def withPartitioning(rewritten: Seq[Transform]): V2CreateTablePlan
}
+trait V2CreateTablePlanX extends LogicalPlan {
+ def name: LogicalPlan
+ def partitioning: Seq[Transform]
+ def tableSchema: StructType
+
+ /**
+ * Creates a copy of this node with the new partitioning transforms. This method is used to
+ * rewrite the partition transforms normalized according to the table schema.
+ */
+ def withPartitioning(rewritten: Seq[Transform]): V2CreateTablePlanX
+}
+
/**
* Create a new table with a v2 catalog.
*/
-case class CreateV2Table(
- catalog: TableCatalog,
- tableName: Identifier,
+case class CreateTable(
+ name: LogicalPlan,
tableSchema: StructType,
partitioning: Seq[Transform],
- properties: Map[String, String],
- ignoreIfExists: Boolean) extends LeafCommand with V2CreateTablePlan {
- override def withPartitioning(rewritten: Seq[Transform]): V2CreateTablePlan = {
+ bucketSpec: Option[BucketSpec],
Review comment:
nit: shall we move `bucketSpec` to `TableSpec`? According to the v2 createTable API, only schema and partitioning are top-level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977611637
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50040/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977587562
**[Test build #145568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145568/testReport)** for PR 34060 at commit [`29ba020`](https://github.com/apache/spark/commit/29ba020e2d3d8da02bb433d85b47a4d117dcce78).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978218475
**[Test build #145591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145591/testReport)** for PR 34060 at commit [`97481b3`](https://github.com/apache/spark/commit/97481b3d4449a042f8d41e17d7fb4ca114a198cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924405406
**[Test build #143477 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143477/testReport)** for PR 34060 at commit [`1965f77`](https://github.com/apache/spark/commit/1965f773825e4f6cd14b71d7f3d32550daaad4c5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924418841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924418841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143477/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-972261870
**[Test build #145346 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145346/testReport)** for PR 34060 at commit [`6ffae91`](https://github.com/apache/spark/commit/6ffae91ee87d74fbd11c4cc62c9282f85a724702).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r753876015
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,21 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
+ val cmd = CreateV2Table(
+ UnresolvedDBObjectName(
+ originalMultipartIdentifier,
+ isNamespace = true),
Review comment:
This should be false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975395328
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49977/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977721539
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145569/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982282199
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982156175
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978727842
**[Test build #145591 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145591/testReport)** for PR 34060 at commit [`97481b3`](https://github.com/apache/spark/commit/97481b3d4449a042f8d41e17d7fb4ca114a198cd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r756898209
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##########
@@ -481,19 +481,26 @@ class PlanResolutionSuite extends AnalysisTest {
"other" -> "20")
parseAndResolve(sql) match {
- case create: CreateV2Table =>
- assert(create.catalog.name == "testcat")
- assert(create.tableName == Identifier.of(Array("mydb"), "table_name"))
+ case create: CatalystCreateTable =>
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].catalog.name == "testcat")
+ assert(create.name.asInstanceOf[ResolvedDBObjectName].nameParts.mkString(".") ==
+ "mydb.table_name")
assert(create.tableSchema == new StructType()
.add("id", LongType)
.add("description", StringType)
.add("point", new StructType().add("x", DoubleType).add("y", DoubleType)))
assert(create.partitioning.isEmpty)
- assert(create.properties == expectedProperties)
Review comment:
I think we don't need to test it now as it's not part of the resolution anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979844619
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978731151
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924297307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r713565908
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##########
@@ -432,6 +432,23 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog {
create.tableSchema.foreach(f => TypeUtils.failWithIntervalType(f.dataType))
+ case create: V2CreateTablePlanX =>
Review comment:
This is only temporary and will be removed after `CreateTableAsSelect`, `ReplaceTable` and `ReplaceTableAsSelect` are migrated to v2 command framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928832541
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-972278961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145346/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928702662
**[Test build #143667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143667/testReport)** for PR 34060 at commit [`9e42d67`](https://github.com/apache/spark/commit/9e42d67de52fa000bed3f21e40527360934a56f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r765992537
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,23 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
- df.schema.asNullable,
- partitioningColumns.getOrElse(Nil).asTransforms.toSeq,
+ val tableProperties = TableSpec(
Review comment:
typo: this should be `tableSpec`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975348840
**[Test build #145505 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145505/testReport)** for PR 34060 at commit [`a2dd853`](https://github.com/apache/spark/commit/a2dd853d3ab8d17cc26d50b183d8363f00ad8bbc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class TableProperties(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-975216033
**[Test build #145505 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145505/testReport)** for PR 34060 at commit [`a2dd853`](https://github.com/apache/spark/commit/a2dd853d3ab8d17cc26d50b183d8363f00ad8bbc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977717376
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978218475
**[Test build #145591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145591/testReport)** for PR 34060 at commit [`97481b3`](https://github.com/apache/spark/commit/97481b3d4449a042f8d41e17d7fb4ca114a198cd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978731151
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145591/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-978279276
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50063/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979771136
**[Test build #145661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145661/testReport)** for PR 34060 at commit [`39320f4`](https://github.com/apache/spark/commit/39320f4f12546475cea54a9914d05973849bb89b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982078391
**[Test build #145733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145733/testReport)** for PR 34060 at commit [`8fdf059`](https://github.com/apache/spark/commit/8fdf059093623b8d36e7231ac539dbb4a6449f40).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982246781
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145733/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34060:
URL: https://github.com/apache/spark/pull/34060
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924180831
**[Test build #143477 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143477/testReport)** for PR 34060 at commit [`1965f77`](https://github.com/apache/spark/commit/1965f773825e4f6cd14b71d7f3d32550daaad4c5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [WIP] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-924297307
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47988/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-936946581
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48401/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928965499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143667/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-977672925
**[Test build #145568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145568/testReport)** for PR 34060 at commit [`29ba020`](https://github.com/apache/spark/commit/29ba020e2d3d8da02bb433d85b47a4d117dcce78).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CreateTable(`
* `case class TableSpec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971944387
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49817/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r755770346
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -819,6 +820,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
redactMapString(map.asCaseSensitiveMap().asScala, maxFields)
case map: Map[_, _] =>
redactMapString(map, maxFields)
+ case t: TableSpec =>
+ redactMapString(t.options, maxFields)
Review comment:
why do we only deal with `t.options` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-972278961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145346/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r753876023
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
##########
@@ -288,19 +289,21 @@ final class DataStreamWriter[T] private[sql](ds: Dataset[T]) {
* Note, currently the new table creation by this API doesn't fully cover the V2 table.
* TODO (SPARK-33638): Full support of v2 table creation
*/
- val cmd = CreateTableStatement(
- originalMultipartIdentifier,
+ val cmd = CreateV2Table(
+ UnresolvedDBObjectName(
+ originalMultipartIdentifier,
+ isNamespace = true),
df.schema.asNullable,
partitioningColumns.getOrElse(Nil).asTransforms.toSeq,
None,
Map.empty[String, String],
- Some(source),
Map.empty[String, String],
- extraOptions.get("path"),
None,
+ extraOptions.get("path"),
None,
+ Some(source),
external = false,
- ifNotExists = false)
+ ignoreIfExists = false)
Review comment:
Yes. We create table if the table doesn't exist, it's the same as we ignore if the table exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-971971611
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49817/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
huaxingao commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r758769208
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -819,6 +820,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with Tre
redactMapString(map.asCaseSensitiveMap().asScala, maxFields)
case map: Map[_, _] =>
redactMapString(map, maxFields)
+ case t: TableSpec =>
+ redactMapString(t.options, maxFields)
+ redactMapString(t.properties, maxFields)
Review comment:
Sorry I didn't get this right. Fixed.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##########
@@ -26,12 +26,12 @@ import org.mockito.invocation.InvocationOnMock
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, EmptyFunctionRegistry, NoSuchTableException, ResolvedFieldName, ResolvedTable, ResolveSessionCatalog, UnresolvedAttribute, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, EmptyFunctionRegistry, NoSuchTableException, ResolvedDBObjectName, ResolvedFieldName, ResolvedTable, ResolveSessionCatalog, UnresolvedAttribute, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat, CatalogTable, CatalogTableType, InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions.{AnsiCast, AttributeReference, EqualTo, Expression, InSubquery, IntegerLiteral, ListQuery, Literal, StringLiteral}
import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
-import org.apache.spark.sql.catalyst.plans.logical.{AlterColumn, AnalysisOnlyCommand, AppendData, Assignment, CreateTableAsSelect, CreateTableStatement, CreateV2Table, DeleteAction, DeleteFromTable, DescribeRelation, DropTable, InsertAction, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, Project, SetTableLocation, SetTableProperties, ShowTableProperties, SubqueryAlias, UnsetTableProperties, UpdateAction, UpdateTable}
+import org.apache.spark.sql.catalyst.plans.logical.{AlterColumn, AnalysisOnlyCommand, AppendData, Assignment, CreateTable => CatalystCreateTable, CreateTableAsSelect, DeleteAction, DeleteFromTable, DescribeRelation, DropTable, InsertAction, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, Project, SetTableLocation, SetTableProperties, ShowTableProperties, SubqueryAlias, UnsetTableProperties, UpdateAction, UpdateTable}
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34060:
URL: https://github.com/apache/spark/pull/34060#discussion_r758286195
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##########
@@ -26,12 +26,12 @@ import org.mockito.invocation.InvocationOnMock
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, EmptyFunctionRegistry, NoSuchTableException, ResolvedFieldName, ResolvedTable, ResolveSessionCatalog, UnresolvedAttribute, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, EmptyFunctionRegistry, NoSuchTableException, ResolvedDBObjectName, ResolvedFieldName, ResolvedTable, ResolveSessionCatalog, UnresolvedAttribute, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat, CatalogTable, CatalogTableType, InMemoryCatalog, SessionCatalog}
import org.apache.spark.sql.catalyst.expressions.{AnsiCast, AttributeReference, EqualTo, Expression, InSubquery, IntegerLiteral, ListQuery, Literal, StringLiteral}
import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
-import org.apache.spark.sql.catalyst.plans.logical.{AlterColumn, AnalysisOnlyCommand, AppendData, Assignment, CreateTableAsSelect, CreateTableStatement, CreateV2Table, DeleteAction, DeleteFromTable, DescribeRelation, DropTable, InsertAction, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, Project, SetTableLocation, SetTableProperties, ShowTableProperties, SubqueryAlias, UnsetTableProperties, UpdateAction, UpdateTable}
+import org.apache.spark.sql.catalyst.plans.logical.{AlterColumn, AnalysisOnlyCommand, AppendData, Assignment, CreateTable => CatalystCreateTable, CreateTableAsSelect, DeleteAction, DeleteFromTable, DescribeRelation, DropTable, InsertAction, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, Project, SetTableLocation, SetTableProperties, ShowTableProperties, SubqueryAlias, UnsetTableProperties, UpdateAction, UpdateTable}
Review comment:
nit: can we change `import org.apache.spark.sql.execution.datasources.CreateTable` to `import org.apache.spark.sql.execution.datasources.CreateTableV1` and keep the `CreateTable` here unchanged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-982078391
**[Test build #145733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145733/testReport)** for PR 34060 at commit [`8fdf059`](https://github.com/apache/spark/commit/8fdf059093623b8d36e7231ac539dbb4a6449f40).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-979826238
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50131/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928702662
**[Test build #143667 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143667/testReport)** for PR 34060 at commit [`9e42d67`](https://github.com/apache/spark/commit/9e42d67de52fa000bed3f21e40527360934a56f7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34060: [SPARK-36850][SQL] Migrate CreateTableStatement to v2 command framework
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34060:
URL: https://github.com/apache/spark/pull/34060#issuecomment-928965499
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143667/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org