You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/15 00:15:39 UTC
[GitHub] [spark] dtenedor opened a new pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
dtenedor opened a new pull request #35855:
URL: https://github.com/apache/spark/pull/35855
### What changes were proposed in this pull request?
Extend CREATE TABLE and REPLACE TABLE statements to support columns with DEFAULT values. Following INSERT INTO statements may then omit the default values or refer to them explicitly with the DEFAULT keyword, in which case the Spark analyzer will automatically insert the appropriate corresponding values in the right places.
Example:
```
CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
INSERT INTO T VALUES (1, 2);
INSERT INTO T VALUES (1, DEFAULT);
INSERT INTO T VALUES (DEFAULT, 6);
SELECT * FROM T;
(1, 2)
(1, 5)
(4, 6)
```
### Why are the changes needed?
This helps users issue INSERT INTO statements with less effort, and helps people creating or updating tables to add custom optional columns for use in specific circumstances as desired.
### How was this patch tested?
This change is covered by new and existing unit test coverage as well as new INSERT INTO query test cases covering a variety of positive and negative scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832161491
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
Review comment:
how about `spark.sql.defaultColumn.enabled`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
Review comment:
Note that `plan.resolveOperatorsWithPruning` will transform all the operators in `i` including query recursively.
Shall we move the methods `addMissingDefaultColumnValues` and `replaceExplicitDefaultColumnValues` to `object ResolveDefaultColumns` so that we don't need to create class `ResolveDefaultColumns` recursively.
The same applies to the following code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834575823
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
Review comment:
This was leftover from before, not needed now. Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srielau commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
srielau commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828188039
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
Review comment:
I just want to clarify that we MUST NOT limit ourselves to constant expressions.
E.g. there are common use cases around e.g. CURRENT_DATE() a well as using NON DETERMINISTIC functions
E.g. to generate UUIDs for primary keys.
We should design accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828206957
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
Review comment:
Hi @srielau, I implemented it this way because the design doc appears to specify that the DEFAULT expression must be folded to a constant [1]. Is this intended? If not, that is OK, I can update this PR to skip the constant-folding and simply validate and store the original text in the column metadata instead.
[1]
<img width="1316" alt="image" src="https://user-images.githubusercontent.com/99207096/158637309-38338720-2d13-4e32-9df1-8d59e2518f5a.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832422708
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
Review comment:
Good question: this is actually the intent to transform recursively, specifically only on the subplan in `i.query`. In this way, the following case matching against `UnresolvedInlineTable` may trigger anywhere that the corresponding `values` list appears in that subplan (e.g. `insert into t select * from values(...)` or more complex cases ending in `values`). I left a comment explaining this here. Otherwise in the more complex cases, the analyzer simply errors that `default` was not found in those cases.
If this SG to you, we can leave it as-is. Otherwise, if you prefer to only match the `i.query` against (`UnresolvedInlineTable` or `Project <- UnresolvedInlineTable`), let me know and I can make that change. We could possibly add another `case` then to `find` if any unresolved `default` references exist in any nested `values`, and return a more specific error message in that case to help guide the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834491135
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
Review comment:
SG, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834427589
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
Review comment:
Let's add a test case for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834476366
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
Review comment:
Done. (My bad, this was leftover from the child PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827996158
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
Review comment:
All the method names should start with lower case words.
`ConstantFoldDefaultExpressions` => `constantFoldDefaultExpressions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828011866
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param insert the INSERT INTO statement to replace explicit DEFAULT column references within.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with DEFAULT column references replaced with their values.
+ */
+ def ReplaceExplicitDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Extract the list of DEFAULT column values from the INSERT INTO statement.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(default) =>
+ parser.parseExpression(f.metadata.getString(default))
+ case _ => Literal(null)
+ }
+ // Handle two types of logical query plans in the target of the INSERT INTO statement:
+ // Inline table: VALUES (0, 1, DEFAULT, ...)
+ // Projection: SELECT 0, 1, DEFAULT, ...
+ // Note that the DEFAULT reference may not participate in complex expressions such as
+ // "DEFAULT + 2"; this generally results in a "not found" error later in the analyzer.
+ val newQuery: LogicalPlan = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == defaultExprs.size) =>
+ val newRows: Seq[Seq[Expression]] =
+ table.rows.map { row: Seq[Expression] =>
+ // Map each row of the VALUES list to its corresponding DEFAULT expression in the
+ // INSERT INTO object table, if the two lists are equal in length.
+ row.zip(defaultExprs).map {
+ case (expr: Expression, defaultExpr: Expression) =>
+ expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(default) => defaultExpr
+ case _ => expr
+ }
+ }
+ }
+ table.copy(rows = newRows)
+ case project: Project if project.projectList.size == defaultExprs.size =>
+ val updated: Seq[NamedExpression] =
+ // Map each expression of the project list to its corresponding DEFAULT expression in the
+ // INSERT INTO object table, if the two lists are equal in length.
+ (project.projectList, defaultExprs, colNames).zipped.map {
+ case (expr: Expression, defaultExpr: Expression, colName: String) =>
+ expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(default) =>
+ Alias(defaultExpr, colName)()
+ case _ => expr
+ }
+ }
+ project.copy(projectList = updated)
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
+ statementType: String):
+ Expression = {
+ try {
+ // Invoke the analyzer over the 'colExpr'.
+ val plan = analyzer.execute(Project(Seq(Alias(colExpr, colName)()), OneRowRelation()))
+ analyzer.checkAnalysis(plan)
+ // Perform constant folding over the result.
+ val folded = ConstantFolding(plan)
+ val result = folded match {
+ case Project(Seq(a: Alias), OneRowRelation()) => a.child
+ }
+ // Make sure the constant folding was successful.
Review comment:
Shall we also check the result data type is compatible with the column type? E.g. `i int default 'abc'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834998874
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.Analyzer
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
Review comment:
Why don't we do constant fold the "current default" as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834428135
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
Review comment:
Why there is ParseException here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r833446908
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog,
+ insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+ case _ if !SQLConf.get.enableDefaultColumns => plan
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ // Create a helper instance of this same rule with the `insert` argument populated as `i`.
+ // Then recursively apply it on the `query` of `i` to transform the result. This recursive
+ // application lets the below case matching against `UnresolvedInlineTable` to trigger
+ // anywhere that operator may appear in the descendants of `i.query`.
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val newQuery = helper.apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
Review comment:
This method is not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832411826
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
Review comment:
SG, that makes the code simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r833459944
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog,
+ insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+ case _ if !SQLConf.get.enableDefaultColumns => plan
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ // Create a helper instance of this same rule with the `insert` argument populated as `i`.
+ // Then recursively apply it on the `query` of `i` to transform the result. This recursive
+ // application lets the below case matching against `UnresolvedInlineTable` to trigger
+ // anywhere that operator may appear in the descendants of `i.query`.
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val newQuery = helper.apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
Review comment:
param schema is not used
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##########
@@ -846,6 +854,238 @@ class InsertSuite extends DataSourceTest with SharedSparkSession {
}
}
+ test("INSERT INTO statements with tables with default columns") {
+ // For most of these cases, the test table 't' has two columns:
+ // (1) name 'i' with boolean type and no default value
+ // (2) name 's' with long integer type and a default value of 42L.
+ //
+ // Positive tests:
+ // When the USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES configuration is enabled, and no
+ // explicit DEFAULT value is available when the INSERT INTO statement provides fewer
+ // values than expected, NULL values are appended in their place.
+ withSQLConf(SQLConf.USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES.key -> "true") {
+ withTable("t") {
+ sql("create table t(i boolean, s bigint) using parquet")
+ sql("insert into t values(true)")
+ checkAnswer(sql("select s from t where i = true"), Seq(null).map(i => Row(i)))
+ }
+ }
+ // The default value for the DEFAULT keyword is the NULL literal.
+ withTable("t") {
+ sql("create table t(i boolean, s bigint) using parquet")
+ sql("insert into t values(true, default)")
+ checkAnswer(sql("select s from t where i = true"), Seq(null).map(i => Row(i)))
+ }
+ // There is a complex expression in the default value.
+ withTable("t") {
+ sql("create table t(i boolean, s string default concat('abc', 'def')) using parquet")
+ sql("insert into t values(true, default)")
+ checkAnswer(sql("select s from t where i = true"), Seq("abcdef").map(i => Row(i)))
+ }
+ // The default value parses correctly and the provided value type is different but coercible.
+ withTable("t") {
+ sql("create table t(i boolean, s bigint default 42) using parquet")
+ sql("insert into t values(false)")
+ checkAnswer(sql("select s from t where i = false"), Seq(42L).map(i => Row(i)))
+ }
+ // There are two trailing default values referenced implicitly by the INSERT INTO statement.
+ withTable("t") {
+ sql("create table t(i int, s bigint default 42, x bigint default 43) using parquet")
+ sql("insert into t values(1)")
+ checkAnswer(sql("select s + x from t where i = 1"), Seq(85L).map(i => Row(i)))
+ }
+ // The table has a partitioning column and a default value is injected.
Review comment:
Do partition columns support default values? If yes, shall we add such a case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1076696649
The work is now split into different PRs:
1. This PR contains only DEFAULT column support for CREATE/REPLACE TABLE statements: https://github.com/apache/spark/pull/35855
2. Separate PR supports DEFAULT columns for INSERT INTO statements: https://github.com/dtenedor/spark/pull/1
3. Third PR adds a query execution class to be used for injecting DEFAULT values during data source scans: https://github.com/apache/spark/pull/35928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r833483971
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##########
@@ -846,6 +854,238 @@ class InsertSuite extends DataSourceTest with SharedSparkSession {
}
}
+ test("INSERT INTO statements with tables with default columns") {
+ // For most of these cases, the test table 't' has two columns:
+ // (1) name 'i' with boolean type and no default value
+ // (2) name 's' with long integer type and a default value of 42L.
+ //
+ // Positive tests:
+ // When the USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES configuration is enabled, and no
+ // explicit DEFAULT value is available when the INSERT INTO statement provides fewer
+ // values than expected, NULL values are appended in their place.
+ withSQLConf(SQLConf.USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES.key -> "true") {
+ withTable("t") {
+ sql("create table t(i boolean, s bigint) using parquet")
+ sql("insert into t values(true)")
+ checkAnswer(sql("select s from t where i = true"), Seq(null).map(i => Row(i)))
+ }
+ }
+ // The default value for the DEFAULT keyword is the NULL literal.
+ withTable("t") {
+ sql("create table t(i boolean, s bigint) using parquet")
+ sql("insert into t values(true, default)")
+ checkAnswer(sql("select s from t where i = true"), Seq(null).map(i => Row(i)))
+ }
+ // There is a complex expression in the default value.
+ withTable("t") {
+ sql("create table t(i boolean, s string default concat('abc', 'def')) using parquet")
+ sql("insert into t values(true, default)")
+ checkAnswer(sql("select s from t where i = true"), Seq("abcdef").map(i => Row(i)))
+ }
+ // The default value parses correctly and the provided value type is different but coercible.
+ withTable("t") {
+ sql("create table t(i boolean, s bigint default 42) using parquet")
+ sql("insert into t values(false)")
+ checkAnswer(sql("select s from t where i = false"), Seq(42L).map(i => Row(i)))
+ }
+ // There are two trailing default values referenced implicitly by the INSERT INTO statement.
+ withTable("t") {
+ sql("create table t(i int, s bigint default 42, x bigint default 43) using parquet")
+ sql("insert into t values(1)")
+ checkAnswer(sql("select s + x from t where i = 1"), Seq(85L).map(i => Row(i)))
+ }
+ // The table has a partitioning column and a default value is injected.
Review comment:
Good question. It looks like the grammar does not support this. I added a test case and initially got a weird parsing error message. I added a rule in the grammar to catch this and return a better error message, and updated the new test case accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834426898
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
Review comment:
Let's add a test case for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834574632
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
Review comment:
Good idea, done. Also the other error cases.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834999055
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.Analyzer
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
Review comment:
nit: let's have a comment on param `analyzer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r835491722
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
##########
@@ -120,6 +121,67 @@ abstract class SessionCatalogSuite extends AnalysisTest with Eventually {
assert(e.contains(s"`$name` is not a valid name for tables/databases."))
}
+ test("create table with default columns") {
+ withBasicCatalog { catalog =>
+ assert(catalog.externalCatalog.listTables("db1").isEmpty)
+ assert(catalog.externalCatalog.listTables("db2").toSet == Set("tbl1", "tbl2"))
+ catalog.createTable(newTable(
+ "tbl3", Some("db1"), defaultColumns = true), ignoreIfExists = false)
+ catalog.createTable(newTable(
+ "tbl3", Some("db2"), defaultColumns = true), ignoreIfExists = false)
+ assert(catalog.externalCatalog.listTables("db1").toSet == Set("tbl3"))
+ assert(catalog.externalCatalog.listTables("db2").toSet == Set("tbl1", "tbl2", "tbl3"))
+ // Inspect the default column values.
+ val db1tbl3 = catalog.externalCatalog.getTable("db1", "tbl3")
+ val currentDefault = ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY
+
+ def findField(name: String, schema: StructType): StructField =
+ schema.fields.filter(_.name == name).head
+ val columnA: StructField = findField("a", db1tbl3.schema)
+ val columnB: StructField = findField("b", db1tbl3.schema)
+ val columnC: StructField = findField("c", db1tbl3.schema)
+ val columnD: StructField = findField("d", db1tbl3.schema)
+ val columnE: StructField = findField("e", db1tbl3.schema)
+
+ val defaultValueColumnA: String = columnA.metadata.getString(currentDefault)
+ val defaultValueColumnB: String = columnB.metadata.getString(currentDefault)
+ val defaultValueColumnC: String = columnC.metadata.getString(currentDefault)
+ val defaultValueColumnD: String = columnD.metadata.getString(currentDefault)
+ val defaultValueColumnE: String = columnE.metadata.getString(currentDefault)
+
+ assert(defaultValueColumnA == "42")
+ assert(defaultValueColumnB == "\"abc\"")
+ assert(defaultValueColumnC == "_@#$%")
+ assert(defaultValueColumnD == "(select min(x) from badtable)")
+ assert(defaultValueColumnE == "41 + 1")
+
+ // Analyze the default column values.
+ val analyzer = new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val statementType = "CREATE TABLE"
+ assert(ResolveDefaultColumns.analyze(analyzer, columnA, statementType).sql == "42")
+ assert(ResolveDefaultColumns.analyze(analyzer, columnB, statementType).sql == "'abc'")
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnC, statementType)
+ }.getMessage.contains("fails to parse as a valid expression"))
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnD, statementType)
+ }.getMessage.contains("fails to resolve as a valid expression"))
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnE, statementType)
+ }.getMessage.contains("statement provided a value of incompatible type"))
+
+ // Make sure that constant-folding default values does not take place when the feature is
+ // disabled.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "false") {
Review comment:
SG, I added that one in the DDLParserSuite as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832407495
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
Review comment:
SG, Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832424326
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -2781,13 +2782,18 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
Option(commentSpec()).map(visitCommentSpec).foreach {
builder.putString("comment", _)
}
+ // Add the 'DEFAULT expression' clause in the column definition, if any, to the column metadata.
+ Option(ctx.defaultExpression()).map(visitDefaultExpression).foreach { field =>
Review comment:
:) I tried this, but it turns out that `getText` strips the spaces. Thus this comment:
```
// Extract the raw expression text so that we can save the user provided text. We don't
// use `Expression.sql` to avoid storing incorrect text caused by bugs in any expression's
// `sql` method. Note: `exprCtx.getText` returns a string without spaces, so we need to
// get the text from the underlying char stream instead.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832437209
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if SQLConf.get.enableDefaultColumns && insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) &&
+ table.rows.exists{_.exists{isExplicitDefaultColumn(_)}}
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
+ schemaWithoutPartitionCols: StructType): Seq[Expression] = {
+ val remainingFields: Seq[StructField] = schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ val numDefaultExprsToAdd: Int = {
+ if (SQLConf.get.useNullsForMissingDefaultColumnValues) {
+ remainingFields.size
+ } else {
+ remainingFields.takeWhile(_.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)).size
+ }
+ }
+ Seq.fill(numDefaultExprsToAdd)(UnresolvedAttribute(CURRENT_DEFAULT_COLUMN_NAME))
+ }
+
+ // Each of the following methods replaces unresolved "DEFAULT" column references with matching
+ // default column values.
+ private def replaceExplicitDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val newRows: Seq[Seq[Expression]] = {
+ table.rows.map { row: Seq[Expression] =>
+ for {
+ i <- 0 until row.size
+ expr = row(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ } yield expr match {
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ defaultExpr
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => expr
+ }
+ }
+ }
+ if (!replaced) return None
+ Some(table.copy(rows = newRows))
+ }
+
+ private def replaceExplicitDefaultColumnValues(project: Project): Option[Project] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val updated: Seq[NamedExpression] = {
+ for {
+ i <- 0 until project.projectList.size
+ projectExpr = project.projectList(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ colName = if (i < colNames.size) colNames(i) else ""
+ } yield projectExpr match {
+ case Alias(u: UnresolvedAttribute, _) if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => projectExpr
+ }
+ }
+ if (!replaced) return None
+ Some(project.copy(projectList = updated))
+ }
+
+ /**
+ * Looks up the schema for the table object of an INSERT INTO statement from the catalog.
+ *
+ * @return the schema of the indicated table, or None if the lookup found nothing.
+ */
+ private def getInsertTableSchema: Option[StructType] = {
+ assert(insert.isDefined)
+ val lookup = try {
+ val tableName = insert.get.table match {
+ case r: UnresolvedRelation => TableIdentifier(r.name)
+ case r: UnresolvedCatalogRelation => r.tableMeta.identifier
+ case _ => return None
+ }
+ catalog.lookupRelation(tableName)
+ } catch {
+ case _: NoSuchTableException => return None
+ }
+ lookup match {
+ case SubqueryAlias(_, r: UnresolvedCatalogRelation) => Some(r.tableMeta.schema)
+ case _ => None
+ }
+ }
+}
+
+/**
+ * This object contains fields to help process DEFAULT columns with the below rule of the same name.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are incomplete.
Review comment:
Yeah, this concept was confusing to me too :) It comes from Serge's design doc [1]. The idea is that in a sequence like:
```
CREATE TABLE t (c1 INT)
INSERT INTO t VALUES (42)
ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
SELECT c1, c2 FROM t
```
In this case, the final query is expected to return 42, 43. The `ALTER TABLE ADD COLUMNS` command executed after there was already data in the table, so in order to enforce this invariant, we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or (2) indicate to each data source that selected columns missing data are to generate the corresponding `DEFAULT` value instead. We choose option (2) for efficiency, and represent this value as the text representation of a folded constant in the `EXISTS_DEFAULT` column metadata.
I put this example in the comment here to help explain it.
[1] https://docs.google.com/document/d/1kNy1YHdVmDGThccFOw-5GROm6gOSf8UF2AWKy_cIAFI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1075440910
(Note: once we are finished with the code review, I can sync and merge the conflicts against head in the `DataSourceV2Strategy` file.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834350327
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
Review comment:
nit: this is not used
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
Review comment:
```suggestion
tableSchema: StructType,
statementType: String): StructType = {
```
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
##########
@@ -1025,16 +1024,30 @@ abstract class CatalogTestUtils {
def newTable(name: String, db: String): CatalogTable = newTable(name, Some(db))
- def newTable(name: String, database: Option[String] = None): CatalogTable = {
+ def newTable(name: String,
+ database: Option[String] = None,
+ defaultColumns: Boolean = false): CatalogTable = {
Review comment:
```suggestion
def newTable(
name: String,
database: Option[String] = None,
defaultColumns: Boolean = false): CatalogTable = {
```
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
Review comment:
```suggestion
field: StructField,
statementType: String,
foldConstants: Boolean = false): Expression = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1076518389
@dtenedor this PR looks good overall.
However, it is quite big and it takes effort to review it carefully.
Do you mind splitting it into two PRs? One for creating tables and another one for inserting into.
I am pretty sure the create table one can be merged very soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834414293
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
Review comment:
Also, the parameter `foldConstants` doesn't seem needed in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1077746518
@dtenedor This one should be ready to me after addressing my comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827997906
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
Review comment:
~~This method is actually duplicated with the optimizer rule `ConstantFolding`. What if we keep the foldable expression in metadata and leave it to `ConstantFolding`?~~
After second thought, we can have better error message if the default value is not foldable or type compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827997906
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
Review comment:
This method is actually duplicated with the optimizer rule `ConstantFolding`. What if we keep the foldable expression in metadata and leave it to `ConstantFolding`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832406161
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
Review comment:
Sure, that SG. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832437876
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r833485406
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog,
+ insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+ case _ if !SQLConf.get.enableDefaultColumns => plan
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ // Create a helper instance of this same rule with the `insert` argument populated as `i`.
+ // Then recursively apply it on the `query` of `i` to transform the result. This recursive
+ // application lets the below case matching against `UnresolvedInlineTable` to trigger
+ // anywhere that operator may appear in the descendants of `i.query`.
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val newQuery = helper.apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1078715590
LGTM overall. This is great work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834476753
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1079221144
Thanks for the review! Looks like this is ready to merge now at your convenience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834419904
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
Review comment:
When will this happen? `EXISTS_DEFAULT_COLUMN_METADATA_KEY` exists and CURRENT_DEFAULT_COLUMN_METADATA_KEY doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834406690
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
##########
@@ -187,12 +190,14 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
RefreshTableExec(r.catalog, r.identifier, recacheTable(r)) :: Nil
case ReplaceTable(ResolvedDBObjectName(catalog, ident), schema, parts, tableSpec, orCreate) =>
+ val newSchema: StructType =
+ ResolveDefaultColumns.constantFoldCurrentDefaultsToExistDefaults(schema, "CREATE TABLE")
Review comment:
I suggest passing the analyzer via function parameter
```
session.sessionState.analyzer
```
instead of creating a new one in object `ResolveDefaultColumns`
After supporting multiple catalogs, different catalogs will have different SQL functions which can be used in default values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827154932
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
Review comment:
nit: please follow the style here: https://github.com/databricks/scala-style-guide#spacing-and-indentation
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
+ statementType: String): Expression = {
+ try {
+ // Invoke the analyzer over the 'colExpr'.
+ val plan = analyzer.execute(Project(Seq(Alias(colExpr, colName)()), OneRowRelation()))
+ analyzer.checkAnalysis(plan)
+ // Perform constant folding over the result.
+ val folded = ConstantFolding(plan)
+ val result = folded match {
+ case Project(Seq(a: Alias), OneRowRelation()) => a.child
+ }
+ // Make sure the constant folding was successful.
+ result match {
+ case _: Literal => result
+ case _ => throw new AnalysisException("non-constant value")
+ }
+ } catch {
+ case ex: AnalysisException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param insert the INSERT INTO statement to replace explicit DEFAULT column references within.
+ * @param catalog session catalog for use when looking up table names for the statement.
+ * @return the updated statement with DEFAULT column references replaced with their values.
+ */
+ def ReplaceExplicitDefaultColumnValues(insert: InsertIntoStatement,
+ catalog: SessionCatalog): InsertIntoStatement = {
Review comment:
nit: 4 spaces indent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832437209
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if SQLConf.get.enableDefaultColumns && insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) &&
+ table.rows.exists{_.exists{isExplicitDefaultColumn(_)}}
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
+ schemaWithoutPartitionCols: StructType): Seq[Expression] = {
+ val remainingFields: Seq[StructField] = schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ val numDefaultExprsToAdd: Int = {
+ if (SQLConf.get.useNullsForMissingDefaultColumnValues) {
+ remainingFields.size
+ } else {
+ remainingFields.takeWhile(_.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)).size
+ }
+ }
+ Seq.fill(numDefaultExprsToAdd)(UnresolvedAttribute(CURRENT_DEFAULT_COLUMN_NAME))
+ }
+
+ // Each of the following methods replaces unresolved "DEFAULT" column references with matching
+ // default column values.
+ private def replaceExplicitDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val newRows: Seq[Seq[Expression]] = {
+ table.rows.map { row: Seq[Expression] =>
+ for {
+ i <- 0 until row.size
+ expr = row(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ } yield expr match {
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ defaultExpr
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => expr
+ }
+ }
+ }
+ if (!replaced) return None
+ Some(table.copy(rows = newRows))
+ }
+
+ private def replaceExplicitDefaultColumnValues(project: Project): Option[Project] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val updated: Seq[NamedExpression] = {
+ for {
+ i <- 0 until project.projectList.size
+ projectExpr = project.projectList(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ colName = if (i < colNames.size) colNames(i) else ""
+ } yield projectExpr match {
+ case Alias(u: UnresolvedAttribute, _) if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => projectExpr
+ }
+ }
+ if (!replaced) return None
+ Some(project.copy(projectList = updated))
+ }
+
+ /**
+ * Looks up the schema for the table object of an INSERT INTO statement from the catalog.
+ *
+ * @return the schema of the indicated table, or None if the lookup found nothing.
+ */
+ private def getInsertTableSchema: Option[StructType] = {
+ assert(insert.isDefined)
+ val lookup = try {
+ val tableName = insert.get.table match {
+ case r: UnresolvedRelation => TableIdentifier(r.name)
+ case r: UnresolvedCatalogRelation => r.tableMeta.identifier
+ case _ => return None
+ }
+ catalog.lookupRelation(tableName)
+ } catch {
+ case _: NoSuchTableException => return None
+ }
+ lookup match {
+ case SubqueryAlias(_, r: UnresolvedCatalogRelation) => Some(r.tableMeta.schema)
+ case _ => None
+ }
+ }
+}
+
+/**
+ * This object contains fields to help process DEFAULT columns with the below rule of the same name.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are incomplete.
Review comment:
Yeah, this concept was confusing to me too :) It comes from Serge's design doc [1]. The idea is that in a sequence like:
```
CREATE TABLE t (c1 INT)
INSERT INTO t VALUES (42)
ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
SELECT c1, c2 FROM t
```
The final query is expected to return 42, 43. The `ALTER TABLE ADD COLUMNS` command executed after there was already data in the table, so in order to enforce this invariant, we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or (2) indicate to each data source that selected columns missing data are to generate the corresponding `DEFAULT` value instead. We choose option (2) for efficiency, and represent this value as the text representation of a folded constant in the `EXISTS_DEFAULT` column metadata.
I put this example in the comment here to help explain it.
[1] https://docs.google.com/document/d/1kNy1YHdVmDGThccFOw-5GROm6gOSf8UF2AWKy_cIAFI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832321011
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if SQLConf.get.enableDefaultColumns && insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) &&
+ table.rows.exists{_.exists{isExplicitDefaultColumn(_)}}
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
+ schemaWithoutPartitionCols: StructType): Seq[Expression] = {
+ val remainingFields: Seq[StructField] = schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ val numDefaultExprsToAdd: Int = {
+ if (SQLConf.get.useNullsForMissingDefaultColumnValues) {
+ remainingFields.size
+ } else {
+ remainingFields.takeWhile(_.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)).size
+ }
+ }
+ Seq.fill(numDefaultExprsToAdd)(UnresolvedAttribute(CURRENT_DEFAULT_COLUMN_NAME))
+ }
+
+ // Each of the following methods replaces unresolved "DEFAULT" column references with matching
+ // default column values.
+ private def replaceExplicitDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val newRows: Seq[Seq[Expression]] = {
+ table.rows.map { row: Seq[Expression] =>
+ for {
+ i <- 0 until row.size
+ expr = row(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ } yield expr match {
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ defaultExpr
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => expr
+ }
+ }
+ }
+ if (!replaced) return None
+ Some(table.copy(rows = newRows))
+ }
+
+ private def replaceExplicitDefaultColumnValues(project: Project): Option[Project] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val updated: Seq[NamedExpression] = {
+ for {
+ i <- 0 until project.projectList.size
+ projectExpr = project.projectList(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ colName = if (i < colNames.size) colNames(i) else ""
+ } yield projectExpr match {
+ case Alias(u: UnresolvedAttribute, _) if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => projectExpr
+ }
+ }
+ if (!replaced) return None
+ Some(project.copy(projectList = updated))
+ }
+
+ /**
+ * Looks up the schema for the table object of an INSERT INTO statement from the catalog.
+ *
+ * @return the schema of the indicated table, or None if the lookup found nothing.
+ */
+ private def getInsertTableSchema: Option[StructType] = {
+ assert(insert.isDefined)
+ val lookup = try {
+ val tableName = insert.get.table match {
+ case r: UnresolvedRelation => TableIdentifier(r.name)
+ case r: UnresolvedCatalogRelation => r.tableMeta.identifier
+ case _ => return None
+ }
+ catalog.lookupRelation(tableName)
+ } catch {
+ case _: NoSuchTableException => return None
+ }
+ lookup match {
+ case SubqueryAlias(_, r: UnresolvedCatalogRelation) => Some(r.tableMeta.schema)
+ case _ => None
+ }
+ }
+}
+
+/**
+ * This object contains fields to help process DEFAULT columns with the below rule of the same name.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are incomplete.
Review comment:
I don't quite understand this. What do you mean by "columns in the source row are incomplete"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832409769
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828278936
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
Review comment:
My bad. Updated the method names. Thank you for your patience, I am new to the style guide :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828283019
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala
##########
@@ -20,21 +20,31 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.catalog.SessionCatalog
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, LocalRelation, LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.types.{StructField, StructType}
/**
* An analyzer rule that replaces [[UnresolvedInlineTable]] with [[LocalRelation]].
*/
-object ResolveInlineTables extends Rule[LogicalPlan] with CastSupport {
+case class ResolveInlineTables(catalog: SessionCatalog) extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
AlwaysProcess.fn, ruleId) {
case table: UnresolvedInlineTable if table.expressionsResolved =>
validateInputDimension(table)
validateInputEvaluable(table)
convert(table)
+ case i @ InsertIntoStatement(_, _, _, (_: UnresolvedInlineTable | _: Project), _, _) =>
Review comment:
I found it was necessary to run this when the inline table is unresolved since at this point, any "default" tokens comprise UnresolvedAttributes which are necessary to replace before we check analysis. I left a comment to this effect at this point in the code to help clarify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828542837
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
Review comment:
This is done, the "current default" value obtained from the initial CREATE/REPLACE TABLE statement is now persisted directly without constant-folding. Instead, a new "exists default" column metadata now holds the constant-folded result separately. Following INSERT/UPDATE/MERGE statements can use the former while table scans can use the latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828005515
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala
##########
@@ -20,21 +20,31 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.catalog.SessionCatalog
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, LocalRelation, LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.types.{StructField, StructType}
/**
* An analyzer rule that replaces [[UnresolvedInlineTable]] with [[LocalRelation]].
*/
-object ResolveInlineTables extends Rule[LogicalPlan] with CastSupport {
+case class ResolveInlineTables(catalog: SessionCatalog) extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
AlwaysProcess.fn, ruleId) {
case table: UnresolvedInlineTable if table.expressionsResolved =>
validateInputDimension(table)
validateInputEvaluable(table)
convert(table)
+ case i @ InsertIntoStatement(_, _, _, (_: UnresolvedInlineTable | _: Project), _, _) =>
+ // This case matches against an INSERT INTO statement whose right-hand side is either a
+ // VALUES list (comprising an inline table) or else a SELECT query (comprising a projection).
+ // In either case, we expect this input to be unresolved at this point because this
+ // [[ResolveInlineTables]] rule runs top-down. It is necessary to match against the entire
+ // INSERT INTO statement in this way in order to provide enough context to know what values
+ // to replace each explicit provided DEFAULT reference with.
+ DefaultColumns.ReplaceExplicitDefaultColumnValues(
Review comment:
How about:
1. filling missing columns with "DEFAULT"
2. replace all the "DEFAULT" as the default column value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827999036
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala
##########
@@ -20,21 +20,31 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.catalog.SessionCatalog
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, LocalRelation, LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.types.{StructField, StructType}
/**
* An analyzer rule that replaces [[UnresolvedInlineTable]] with [[LocalRelation]].
*/
-object ResolveInlineTables extends Rule[LogicalPlan] with CastSupport {
+case class ResolveInlineTables(catalog: SessionCatalog) extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
AlwaysProcess.fn, ruleId) {
case table: UnresolvedInlineTable if table.expressionsResolved =>
validateInputDimension(table)
validateInputEvaluable(table)
convert(table)
+ case i @ InsertIntoStatement(_, _, _, (_: UnresolvedInlineTable | _: Project), _, _) =>
Review comment:
Shall we only run the following when i.table is resolved? Thus we don't need to handle `UnresolvedInlineTable` in this rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r835449248
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.Analyzer
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
Review comment:
Good question. This is the behavior Serge requested in his specification.
The reason for constant-folding the EXISTS_DEFAULT is to make the end-user visible behavior the same, after executing an ALTER TABLE ADD COLUMNS command with DEFAULT value, as if the system had performed an exhaustive backfill of the provided value to all previously existing rows in the table instead. We choose to avoid doing such a backfill because it would be a time-consuming and costly operation. Instead, we elect to store the EXISTS_DEFAULT in the column metadata for future reference when querying data out of the data source. In turn, each data source then takes responsibility to provide the constant-folded value in the EXISTS_DEFAULT metadata for such columns where the value is not present in storage.
I have added this explanation to the comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832409268
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
+ .internal()
+ .doc("When true, and DEFAULT columns are enabled, allow column definitions lacking " +
+ "explicit default values to behave as if they had specified DEFAULT NULL instead. " +
+ "For example, this allows most INSERT INTO statements to specify only a prefix of the " +
+ "columns in the target table, and the remaining columns will receive NULL values.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
Yeah this PR is already big :) we can leave this disabled by default for now, and then consider next whether we want to enable it. Without enabling this by default, this new functionality in this PR does not change the results of any INSERT INTO statements where the corresponding CREATE/REPLACE TABLE commands lack any DEFAULT values (which should be all the cases up until now).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832407128
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
Review comment:
Done (and for the USE_NULLS_... config below as well.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827571310
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828313396
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala
##########
@@ -20,21 +20,31 @@ package org.apache.spark.sql.catalyst.analysis
import scala.util.control.NonFatal
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.catalog.SessionCatalog
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, LocalRelation, LogicalPlan, Project}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.AlwaysProcess
import org.apache.spark.sql.types.{StructField, StructType}
/**
* An analyzer rule that replaces [[UnresolvedInlineTable]] with [[LocalRelation]].
*/
-object ResolveInlineTables extends Rule[LogicalPlan] with CastSupport {
+case class ResolveInlineTables(catalog: SessionCatalog) extends Rule[LogicalPlan] with CastSupport {
override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
AlwaysProcess.fn, ruleId) {
case table: UnresolvedInlineTable if table.expressionsResolved =>
validateInputDimension(table)
validateInputEvaluable(table)
convert(table)
+ case i @ InsertIntoStatement(_, _, _, (_: UnresolvedInlineTable | _: Project), _, _) =>
+ // This case matches against an INSERT INTO statement whose right-hand side is either a
+ // VALUES list (comprising an inline table) or else a SELECT query (comprising a projection).
+ // In either case, we expect this input to be unresolved at this point because this
+ // [[ResolveInlineTables]] rule runs top-down. It is necessary to match against the entire
+ // INSERT INTO statement in this way in order to provide enough context to know what values
+ // to replace each explicit provided DEFAULT reference with.
+ DefaultColumns.ReplaceExplicitDefaultColumnValues(
Review comment:
Good idea, that simplifies things :) done.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param insert the INSERT INTO statement to replace explicit DEFAULT column references within.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with DEFAULT column references replaced with their values.
+ */
+ def ReplaceExplicitDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Extract the list of DEFAULT column values from the INSERT INTO statement.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(default) =>
+ parser.parseExpression(f.metadata.getString(default))
+ case _ => Literal(null)
+ }
+ // Handle two types of logical query plans in the target of the INSERT INTO statement:
+ // Inline table: VALUES (0, 1, DEFAULT, ...)
+ // Projection: SELECT 0, 1, DEFAULT, ...
+ // Note that the DEFAULT reference may not participate in complex expressions such as
+ // "DEFAULT + 2"; this generally results in a "not found" error later in the analyzer.
+ val newQuery: LogicalPlan = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == defaultExprs.size) =>
+ val newRows: Seq[Seq[Expression]] =
+ table.rows.map { row: Seq[Expression] =>
+ // Map each row of the VALUES list to its corresponding DEFAULT expression in the
+ // INSERT INTO object table, if the two lists are equal in length.
+ row.zip(defaultExprs).map {
+ case (expr: Expression, defaultExpr: Expression) =>
+ expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(default) => defaultExpr
+ case _ => expr
+ }
+ }
+ }
+ table.copy(rows = newRows)
+ case project: Project if project.projectList.size == defaultExprs.size =>
+ val updated: Seq[NamedExpression] =
+ // Map each expression of the project list to its corresponding DEFAULT expression in the
+ // INSERT INTO object table, if the two lists are equal in length.
+ (project.projectList, defaultExprs, colNames).zipped.map {
+ case (expr: Expression, defaultExpr: Expression, colName: String) =>
+ expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(default) =>
+ Alias(defaultExpr, colName)()
+ case _ => expr
+ }
+ }
+ project.copy(projectList = updated)
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
+ statementType: String):
+ Expression = {
+ try {
+ // Invoke the analyzer over the 'colExpr'.
+ val plan = analyzer.execute(Project(Seq(Alias(colExpr, colName)()), OneRowRelation()))
+ analyzer.checkAnalysis(plan)
+ // Perform constant folding over the result.
+ val folded = ConstantFolding(plan)
+ val result = folded match {
+ case Project(Seq(a: Alias), OneRowRelation()) => a.child
+ }
+ // Make sure the constant folding was successful.
Review comment:
I was doing this for the INSERT INTO statements but not CREATE/REPLACE TABLE statements during constant-folding. This is a good point, I moved the type coercion logic into the 'analyze' helper method so it always runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832316164
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##########
@@ -2781,13 +2782,18 @@ class AstBuilder extends SqlBaseParserBaseVisitor[AnyRef] with SQLConfHelper wit
Option(commentSpec()).map(visitCommentSpec).foreach {
builder.putString("comment", _)
}
+ // Add the 'DEFAULT expression' clause in the column definition, if any, to the column metadata.
+ Option(ctx.defaultExpression()).map(visitDefaultExpression).foreach { field =>
Review comment:
nit: We don't need function visitDefaultExpression. We can define a variable in the grammar file like
```
defaultExpression
: DEFAULT defaultValue=expression
```
and then we can have
```
Option(ctx.defaultExpression()).map(_.defaultValue.getText)
```
here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834386995
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
Review comment:
This method is actually called from `DataSourceV2Strategy`.
Shall we move the whole object under `org.apache.spark.sql.catalyst.util` or some where else, instead of `org.apache.spark.sql.catalyst.analysis`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
Review comment:
This method is actually called from `DataSourceV2Strategy`.
Shall we move the whole object under `org.apache.spark.sql.catalyst.util` or somewhere else, instead of `org.apache.spark.sql.catalyst.analysis`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832423439
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
##########
@@ -1025,16 +1024,29 @@ abstract class CatalogTestUtils {
def newTable(name: String, db: String): CatalogTable = newTable(name, Some(db))
- def newTable(name: String, database: Option[String] = None): CatalogTable = {
+ def newTable(name: String, database: Option[String] = None,
+ defaultColumns: Boolean = false): CatalogTable = {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832407595
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
Review comment:
SG, Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834477457
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1068090896
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang closed pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #35855:
URL: https://github.com/apache/spark/pull/35855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r835479625
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
##########
@@ -120,6 +121,67 @@ abstract class SessionCatalogSuite extends AnalysisTest with Eventually {
assert(e.contains(s"`$name` is not a valid name for tables/databases."))
}
+ test("create table with default columns") {
+ withBasicCatalog { catalog =>
+ assert(catalog.externalCatalog.listTables("db1").isEmpty)
+ assert(catalog.externalCatalog.listTables("db2").toSet == Set("tbl1", "tbl2"))
+ catalog.createTable(newTable(
+ "tbl3", Some("db1"), defaultColumns = true), ignoreIfExists = false)
+ catalog.createTable(newTable(
+ "tbl3", Some("db2"), defaultColumns = true), ignoreIfExists = false)
+ assert(catalog.externalCatalog.listTables("db1").toSet == Set("tbl3"))
+ assert(catalog.externalCatalog.listTables("db2").toSet == Set("tbl1", "tbl2", "tbl3"))
+ // Inspect the default column values.
+ val db1tbl3 = catalog.externalCatalog.getTable("db1", "tbl3")
+ val currentDefault = ResolveDefaultColumns.CURRENT_DEFAULT_COLUMN_METADATA_KEY
+
+ def findField(name: String, schema: StructType): StructField =
+ schema.fields.filter(_.name == name).head
+ val columnA: StructField = findField("a", db1tbl3.schema)
+ val columnB: StructField = findField("b", db1tbl3.schema)
+ val columnC: StructField = findField("c", db1tbl3.schema)
+ val columnD: StructField = findField("d", db1tbl3.schema)
+ val columnE: StructField = findField("e", db1tbl3.schema)
+
+ val defaultValueColumnA: String = columnA.metadata.getString(currentDefault)
+ val defaultValueColumnB: String = columnB.metadata.getString(currentDefault)
+ val defaultValueColumnC: String = columnC.metadata.getString(currentDefault)
+ val defaultValueColumnD: String = columnD.metadata.getString(currentDefault)
+ val defaultValueColumnE: String = columnE.metadata.getString(currentDefault)
+
+ assert(defaultValueColumnA == "42")
+ assert(defaultValueColumnB == "\"abc\"")
+ assert(defaultValueColumnC == "_@#$%")
+ assert(defaultValueColumnD == "(select min(x) from badtable)")
+ assert(defaultValueColumnE == "41 + 1")
+
+ // Analyze the default column values.
+ val analyzer = new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val statementType = "CREATE TABLE"
+ assert(ResolveDefaultColumns.analyze(analyzer, columnA, statementType).sql == "42")
+ assert(ResolveDefaultColumns.analyze(analyzer, columnB, statementType).sql == "'abc'")
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnC, statementType)
+ }.getMessage.contains("fails to parse as a valid expression"))
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnD, statementType)
+ }.getMessage.contains("fails to resolve as a valid expression"))
+ assert(intercept[AnalysisException] {
+ ResolveDefaultColumns.analyze(analyzer, columnE, statementType)
+ }.getMessage.contains("statement provided a value of incompatible type"))
+
+ // Make sure that constant-folding default values does not take place when the feature is
+ // disabled.
+ withSQLConf(SQLConf.ENABLE_DEFAULT_COLUMNS.key -> "false") {
Review comment:
nit: actually I was expecting a test case which captures parser exception when the conf is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834507653
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
Review comment:
Good point. This should really just assume that `CURRENT_DEFAULT_COLUMN_METADATA_KEY` is populated. I updated the call site to check that explicitly and then deduplicated the two extractions of the column metadata here in this `analyze` method into one, which can skip the presence check and just extract the metadata value directly now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834478153
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
##########
@@ -1025,16 +1024,30 @@ abstract class CatalogTestUtils {
def newTable(name: String, db: String): CatalogTable = newTable(name, Some(db))
- def newTable(name: String, database: Option[String] = None): CatalogTable = {
+ def newTable(name: String,
+ database: Option[String] = None,
+ defaultColumns: Boolean = false): CatalogTable = {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r835449248
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.Analyzer
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
Review comment:
Good question.
The reason for constant-folding the EXISTS_DEFAULT is to make the end-user visible behavior the same after executing an ALTER TABLE ADD COLUMNS command with DEFAULT value as if the system had performed an exhaustive backfill of the provided value to all previously existing rows in the table instead. We choose to avoid doing that because it is a time-consuming and costly operation, electing instead to store the EXISTS_DEFAULT in the column metadata for future reference when querying data out of the data source. Each data source takes responsibility to provide the constant-folded value in the EXISTS_DEFAULT metadata for such columns where the value is not present in storage.
I have added this explanation to the comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834868656
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2805,6 +2805,17 @@ object SQLConf {
.booleanConf
.createWithDefault(sys.env.get("SPARK_ANSI_SQL_MODE").contains("true"))
+ val ENABLE_DEFAULT_COLUMNS =
Review comment:
We need a test case with this conf as `false`, too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1076553634
> @dtenedor this PR looks good overall.
However, it is quite big and it takes effort to review it carefully.
Do you mind splitting it into two PRs? One for creating tables and another one for inserting into.
I am pretty sure the create table one can be merged very soon.
SG, this is a good way to split the changes. I can also take this chance to sync the latest changes from master which should remove the 'This branch has conflicts that must be resolved' message. Then we can get the automatic tests to cover each of the two changes as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r827571422
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses DEFAULT column text to an expression, returning a reasonable error upon failure.
+ *
+ * @param colName the name of the DEFAULT column whose text we endeavor to parse.
+ * @param colText the string contents of the DEFAULT column value.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return the expression resulting from the parsing step.
+ */
+ private def Parse(colName: String, colText: String, statementType: String): Expression = {
+ try {
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Analyzes and constant-folds `colExpr`, returning a reasonable error message upon failure.
+ *
+ * @param colExpr result of a parsing operation suitable for consumption by analysis.
+ * @param colText string contents of the DEFAULT column value; useful for errors.
+ * @param colName string name of the DEFAULT column; useful for errors.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def Analyze(colExpr: Expression, colText: String, colName: String,
+ statementType: String): Expression = {
+ try {
+ // Invoke the analyzer over the 'colExpr'.
+ val plan = analyzer.execute(Project(Seq(Alias(colExpr, colName)()), OneRowRelation()))
+ analyzer.checkAnalysis(plan)
+ // Perform constant folding over the result.
+ val folded = ConstantFolding(plan)
+ val result = folded match {
+ case Project(Seq(a: Alias), OneRowRelation()) => a.child
+ }
+ // Make sure the constant folding was successful.
+ result match {
+ case _: Literal => result
+ case _ => throw new AnalysisException("non-constant value")
+ }
+ } catch {
+ case ex: AnalysisException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ colName + analysisPrefix + s"$colText yields ${ex.getMessage}")
+ }
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param insert the INSERT INTO statement to replace explicit DEFAULT column references within.
+ * @param catalog session catalog for use when looking up table names for the statement.
+ * @return the updated statement with DEFAULT column references replaced with their values.
+ */
+ def ReplaceExplicitDefaultColumnValues(insert: InsertIntoStatement,
+ catalog: SessionCatalog): InsertIntoStatement = {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832294357
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
##########
@@ -1025,16 +1024,29 @@ abstract class CatalogTestUtils {
def newTable(name: String, db: String): CatalogTable = newTable(name, Some(db))
- def newTable(name: String, database: Option[String] = None): CatalogTable = {
+ def newTable(name: String, database: Option[String] = None,
+ defaultColumns: Boolean = false): CatalogTable = {
Review comment:
```suggestion
def newTable(
name: String,
database: Option[String] = None,
defaultColumns: Boolean = false): CatalogTable = {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832437209
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if SQLConf.get.enableDefaultColumns &&
+ i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ val newQuery = ResolveDefaultColumns(catalog, Some(i)).apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if SQLConf.get.enableDefaultColumns && insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if SQLConf.get.enableDefaultColumns && project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) &&
+ table.rows.exists{_.exists{isExplicitDefaultColumn(_)}}
+
+ // Each of the following methods adds a projection over the input plan to generate missing default
+ // column values.
+ private def addMissingDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val numQueryOutputs: Int = table.rows(0).size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ if (newDefaultExprs.isEmpty) return None
+ Some(table.copy(names = table.names ++ newNames,
+ rows = table.rows.map { row => row ++ newDefaultExprs }))
+ }
+
+ private def addMissingDefaultColumnValues(project: Project): Option[Project] = {
+ val numQueryOutputs: Int = project.projectList.size
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val newDefaultExprs: Seq[Expression] =
+ getDefaultExprs(numQueryOutputs, schema, schemaWithoutPartitionCols)
+ val newAliases: Seq[NamedExpression] =
+ newDefaultExprs.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)()
+ }
+ if (newDefaultExprs.isEmpty) return None
+ Some(project.copy(projectList = project.projectList ++ newAliases))
+ }
+
+ // This is a helper for the addMissingDefaultColumnValues methods above.
+ private def getDefaultExprs(numQueryOutputs: Int, schema: StructType,
+ schemaWithoutPartitionCols: StructType): Seq[Expression] = {
+ val remainingFields: Seq[StructField] = schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ val numDefaultExprsToAdd: Int = {
+ if (SQLConf.get.useNullsForMissingDefaultColumnValues) {
+ remainingFields.size
+ } else {
+ remainingFields.takeWhile(_.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)).size
+ }
+ }
+ Seq.fill(numDefaultExprsToAdd)(UnresolvedAttribute(CURRENT_DEFAULT_COLUMN_NAME))
+ }
+
+ // Each of the following methods replaces unresolved "DEFAULT" column references with matching
+ // default column values.
+ private def replaceExplicitDefaultColumnValues(
+ table: UnresolvedInlineTable): Option[UnresolvedInlineTable] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val newRows: Seq[Seq[Expression]] = {
+ table.rows.map { row: Seq[Expression] =>
+ for {
+ i <- 0 until row.size
+ expr = row(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ } yield expr match {
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ defaultExpr
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => expr
+ }
+ }
+ }
+ if (!replaced) return None
+ Some(table.copy(rows = newRows))
+ }
+
+ private def replaceExplicitDefaultColumnValues(project: Project): Option[Project] = {
+ assert(insert.isDefined)
+ val schema: StructType = getInsertTableSchema.getOrElse(return None)
+ val schemaWithoutPartitionCols =
+ StructType(schema.fields.dropRight(insert.get.partitionSpec.size))
+ val colNames: Seq[String] = schemaWithoutPartitionCols.fields.map { _.name }
+ val defaultExprs: Seq[Expression] = schemaWithoutPartitionCols.fields.map {
+ case f if f.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY) => analyze(f, "INSERT")
+ case _ => Literal(null)
+ }
+ var replaced = false
+ val updated: Seq[NamedExpression] = {
+ for {
+ i <- 0 until project.projectList.size
+ projectExpr = project.projectList(i)
+ defaultExpr = if (i < defaultExprs.size) defaultExprs(i) else Literal(null)
+ colName = if (i < colNames.size) colNames(i) else ""
+ } yield projectExpr match {
+ case Alias(u: UnresolvedAttribute, _) if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case u: UnresolvedAttribute if isExplicitDefaultColumn(u) =>
+ replaced = true
+ Alias(defaultExpr, colName)()
+ case expr@_ if expr.find{isExplicitDefaultColumn}.isDefined =>
+ // Return a more descriptive error message if the user tries to nest the DEFAULT column
+ // reference inside some other expression, such as DEFAULT + 1 (this is not allowed).
+ throw new AnalysisException(DEFAULTS_IN_EXPRESSIONS_ERROR)
+ case _ => projectExpr
+ }
+ }
+ if (!replaced) return None
+ Some(project.copy(projectList = updated))
+ }
+
+ /**
+ * Looks up the schema for the table object of an INSERT INTO statement from the catalog.
+ *
+ * @return the schema of the indicated table, or None if the lookup found nothing.
+ */
+ private def getInsertTableSchema: Option[StructType] = {
+ assert(insert.isDefined)
+ val lookup = try {
+ val tableName = insert.get.table match {
+ case r: UnresolvedRelation => TableIdentifier(r.name)
+ case r: UnresolvedCatalogRelation => r.tableMeta.identifier
+ case _ => return None
+ }
+ catalog.lookupRelation(tableName)
+ } catch {
+ case _: NoSuchTableException => return None
+ }
+ lookup match {
+ case SubqueryAlias(_, r: UnresolvedCatalogRelation) => Some(r.tableMeta.schema)
+ case _ => None
+ }
+ }
+}
+
+/**
+ * This object contains fields to help process DEFAULT columns with the below rule of the same name.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are incomplete.
Review comment:
Yeah, this concept was confusing to me too :) It comes from Serge's design doc [1]. The idea is that in a sequence like this:
```
CREATE TABLE t (c1 INT)
INSERT INTO t VALUES (42)
ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
SELECT c1, c2 FROM t
```
The final query is expected to return 42, 43. The `ALTER TABLE ADD COLUMNS` command executed after there was already data in the table, so in order to enforce this invariant, we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or (2) indicate to each data source that selected columns missing data are to generate the corresponding `DEFAULT` value instead. We choose option (2) for efficiency, and represent this value as the text representation of a folded constant in the `EXISTS_DEFAULT` column metadata.
I put this example in the comment here to help explain it.
[1] https://docs.google.com/document/d/1kNy1YHdVmDGThccFOw-5GROm6gOSf8UF2AWKy_cIAFI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832412059
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
Review comment:
SG, that makes the code simpler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832409955
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r833458194
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,371 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog,
+ insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
+ AlwaysProcess.fn, ruleId) {
+ case _ if !SQLConf.get.enableDefaultColumns => plan
+
+ case i@InsertIntoStatement(_, _, _, _, _, _)
+ if i.query.collectFirst { case u: UnresolvedInlineTable => u }.isDefined =>
+ // Create a helper instance of this same rule with the `insert` argument populated as `i`.
+ // Then recursively apply it on the `query` of `i` to transform the result. This recursive
+ // application lets the below case matching against `UnresolvedInlineTable` to trigger
+ // anywhere that operator may appear in the descendants of `i.query`.
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val newQuery = helper.apply(i.query)
+ i.copy(query = newQuery)
+
+ case table: UnresolvedInlineTable
+ if insert.isDefined &&
+ table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ val expanded: UnresolvedInlineTable = addMissingDefaultColumnValues(table).getOrElse(table)
+ replaceExplicitDefaultColumnValues(expanded).getOrElse(table)
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if !project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(project)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+
+ case i@InsertIntoStatement(_, _, _, project: Project, _, _)
+ if project.resolved =>
+ val helper = ResolveDefaultColumns(catalog, Some(i))
+ val expanded: Project = helper.addMissingDefaultColumnValues(project).getOrElse(project)
+ val replaced: Option[Project] = helper.replaceExplicitDefaultColumnValues(expanded)
+ if (replaced.isDefined) i.copy(query = replaced.get) else i
+ }
+
+ // Helper method to check if an expression is an explicit DEFAULT column reference.
+ private def isExplicitDefaultColumn(expr: Expression): Boolean = expr match {
+ case u: UnresolvedAttribute if u.name.equalsIgnoreCase(CURRENT_DEFAULT_COLUMN_NAME) => true
+ case _ => false
+ }
+
+ // Helper method to check that an inline table has any explicit DEFAULT column references.
+ private def hasExplicitDefaultReferences(table: UnresolvedInlineTable): Boolean =
Review comment:
Good catch, removed this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r832161821
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
Review comment:
3.4.0
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
+ .internal()
+ .doc("When true, and DEFAULT columns are enabled, allow column definitions lacking " +
+ "explicit default values to behave as if they had specified DEFAULT NULL instead. " +
+ "For example, this allows most INSERT INTO statements to specify only a prefix of the " +
+ "columns in the target table, and the remaining columns will receive NULL values.")
+ .version("3.3.0")
Review comment:
3.4.0
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
Review comment:
3.4.0
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
Review comment:
how about `spark.sql.defaultColumn.useNullsForMissingDefaultValues`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
Review comment:
how about `spark.sql.defaultColumn.useNullsForMissingDefaultValues`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
+ .internal()
+ .doc("When true, and DEFAULT columns are enabled, allow column definitions lacking " +
+ "explicit default values to behave as if they had specified DEFAULT NULL instead. " +
+ "For example, this allows most INSERT INTO statements to specify only a prefix of the " +
+ "columns in the target table, and the remaining columns will receive NULL values.")
+ .version("3.3.0")
Review comment:
3.4.0
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
+ .internal()
+ .doc("When true, and DEFAULT columns are enabled, allow column definitions lacking " +
+ "explicit default values to behave as if they had specified DEFAULT NULL instead. " +
+ "For example, this allows most INSERT INTO statements to specify only a prefix of the " +
+ "columns in the target table, and the remaining columns will receive NULL values.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
+1 for making the default value false
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
Review comment:
```suggestion
catalog: SessionCatalog,
insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
```
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
Review comment:
nit: Let's check the conf enableDefaultColumns at very beginning:
```
if (!conf.enableDefaultColumns) {
return plan
}
```
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
+ .internal()
+ .doc("When true, allow CREATE TABLE, REPLACE TABLE, and ALTER COLUMN statements to set or " +
+ "update default values for specific columns. Following INSERT, MERGE, and UPDATE " +
+ "statements may then omit these values and their values will be injected automatically " +
+ "instead.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(true)
+
+ val USE_NULLS_FOR_MISSING_DEFAULT_COLUMN_VALUES =
+ buildConf("spark.sql.parser.useNullsForMissingDefaultColumnValues")
+ .internal()
+ .doc("When true, and DEFAULT columns are enabled, allow column definitions lacking " +
+ "explicit default values to behave as if they had specified DEFAULT NULL instead. " +
+ "For example, this allows most INSERT INTO statements to specify only a prefix of the " +
+ "columns in the target table, and the remaining columns will receive NULL values.")
+ .version("3.3.0")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
+1 for making the default value false
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2709,6 +2709,28 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ENABLE_DEFAULT_COLUMNS =
+ buildConf("spark.sql.parser.enableDefaultColumns")
Review comment:
how about `spark.sql.defaultColumn.enabled`?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsWithPruning(
Review comment:
nit: Let's check the conf enableDefaultColumns at very beginning:
```
if (!conf.enableDefaultColumns) {
return plan
}
```
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,354 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.ResolveDefaultColumns._
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This is a rule to process DEFAULT columns in statements such as CREATE/REPLACE TABLE.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
+ * INSERT INTO T VALUES (1, 2);
+ * INSERT INTO T VALUES (1, DEFAULT);
+ * INSERT INTO T VALUES (DEFAULT, 6);
+ * SELECT * FROM T;
+ * (1, 2)
+ * (1, 5)
+ * (4, 6)
+ *
+ * @param catalog the catalog to use for looking up the schema of INSERT INTO table objects.
+ * @param insert the enclosing INSERT statement for which this rule is processing the query, if any.
+ */
+case class ResolveDefaultColumns(
+ catalog: SessionCatalog, insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
Review comment:
```suggestion
catalog: SessionCatalog,
insert: Option[InsertIntoStatement] = None) extends Rule[LogicalPlan] {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE and INSERT INTO statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r828411289
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DefaultColumns.scala
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions.{Expression, _}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.types._
+
+/**
+ * This class contains logic for processing DEFAULT columns in statements such as CREATE TABLE.
+ */
+object DefaultColumns {
+ val default = "default"
+ val analysisPrefix =
+ " has a DEFAULT value which fails to resolve to a valid constant expression: "
+ val columnDefaultNotFound = "Column 'default' does not exist"
+ lazy val parser = new CatalystSqlParser()
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+
+ /**
+ * Finds DEFAULT expressions in CREATE/REPLACE TABLE commands and constant-folds then.
+ *
+ * Example:
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5) becomes
+ * CREATE TABLE T(a INT, b INT DEFAULT 10)
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def ConstantFoldDefaultExpressions(tableSchema: StructType, statementType: String): StructType = {
+ // Get the list of column indexes in the CREATE TABLE command with DEFAULT values.
+ val (fields: Array[StructField], indexes: Array[Int]) =
+ tableSchema.fields.zipWithIndex.filter { case (f, i) => f.metadata.contains(default) }.unzip
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val colNames: Seq[String] = fields.map { _.name }
+ val defaults: Seq[String] = fields.map { _.metadata.getString(default) }
+ // Extract the list of DEFAULT column values from the CreateTable command.
+ val exprs: Seq[Expression] = colNames.zip(defaults).map {
+ case (name, text) => Parse(name, text, statementType)
+ }
+ // Analyze and constant-fold each parse result.
+ val analyzed: Seq[Expression] = (exprs, defaults, colNames).zipped.map {
+ case (expr, default, name) => Analyze(expr, default, name, statementType)
+ }
+ // Create a map from the column index of each DEFAULT column to its type.
+ val indexMap: Map[Int, StructField] = (indexes, fields, analyzed).zipped.map {
+ case (index, field, expr) =>
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(default, expr.sql).build()
+ (index, field.copy(metadata = newMetadata))
+ }.toMap
+ // Finally, replace the original struct fields with the new ones.
+ val newFields: Seq[StructField] =
+ tableSchema.fields.zipWithIndex.map { case (f, i) => indexMap.getOrElse(i, f) }
+ StructType(newFields)
+ }
+
+ /**
+ * Adds a projection over the plan in `insert.query` generating missing default column values.
+ *
+ * @param insert the INSERT INTO statement to add missing DEFAULT column references to.
+ * @param catalog the catalog to use for looking up the schema of the INSERT INTO table object.
+ * @return the updated statement with missing DEFAULT column values appended to the list.
+ */
+ def AddProjectionForMissingDefaultColumnValues(
+ insert: InsertIntoStatement, catalog: SessionCatalog): InsertIntoStatement = {
+ // Compute the number of attributes returned by the INSERT INTO statement.
+ val numQueryOutputs: Int = insert.query match {
+ case table: UnresolvedInlineTable
+ if table.rows.nonEmpty && table.rows.forall(_.size == table.rows(0).size) =>
+ table.rows(0).size
+ case project: Project => project.projectList.size
+ case _ => return insert
+ }
+ // The table value provides the DEFAULT column values as text; analyze them into expressions.
+ val schema: StructType = getInsertTableSchema(insert, catalog).getOrElse(return insert)
+ val schemaWithoutPartitionCols = StructType(schema.fields.dropRight(insert.partitionSpec.size))
+ val coerced: Seq[Expression] = for {
+ field <- schemaWithoutPartitionCols.fields.drop(numQueryOutputs)
+ name: String = field.name
+ text: String =
+ if (field.metadata.contains(default)) field.metadata.getString(default) else "NULL"
+ // Parse the DEFAULT column expression. If the parsing fails, throw an error to the user.
+ expr: Expression = Parse(name, text, "INSERT")
+ // Analyze and constant-fold each result.
+ analyzed: Expression = Analyze(expr, text, name, "INSERT")
+ // Perform implicit coercion from the provided expression type to the required column type.
+ errorPrefix = "Failed to execute INSERT command because the destination table column "
+ coerced: Expression =
+ if (field.dataType == analyzed.dataType) {
+ analyzed
+ } else if (Cast.canUpCast(analyzed.dataType, field.dataType)) {
+ Cast(analyzed, field.dataType)
+ } else {
+ throw new AnalysisException(errorPrefix +
+ s"$name has a DEFAULT value with type ${field.dataType}, but the " +
+ s"query provided a value of incompatible type ${analyzed.dataType}")
+ }
+ } yield coerced
+ // Finally, return a projection of the original `insert.query` output attributes plus new
+ // aliases over the DEFAULT column values.
+ // If the insertQuery is an existing Project, flatten them together.
+ val newQuery = insert.query match {
+ case Project(projectList, child) =>
+ val newAliases: Seq[NamedExpression] = coerced.zip(schemaWithoutPartitionCols.fields).map {
+ case (expr, field) => Alias(expr, field.name)() }
+ Project(projectList ++ newAliases, child)
+ case table: UnresolvedInlineTable =>
+ val newNames: Seq[String] =
+ schemaWithoutPartitionCols.fields.drop(numQueryOutputs).map { _.name }
+ table.copy(names = table.names ++ newNames, rows = table.rows.map { row => row ++ coerced })
+ case _ => insert.query
+ }
+ insert.copy(query = newQuery)
+ }
+
+ /**
+ * Replaces unresolved "DEFAULT" column references with matching default column values.
+ *
+ * Background: CREATE TABLE and ALTER TABLE invocations support setting column default values for
+ * later operations. Following INSERT, and INSERT MERGE commands may then reference the value
+ * using the DEFAULT keyword as needed.
+ *
+ * Example:
+ * CREATE TABLE T(a INT DEFAULT 4, b INT NOT NULL DEFAULT 5);
Review comment:
Resolved this confusion offline with Serge. This should be evaluated at the INSERT INTO time:
```
Catalyst
The catalyst design is straight forward. The defaults are read in with the rest of the column info.
We leave DEFAULT keywords at DML as unresolved until analysis.
```
I can update the PR accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #35855:
URL: https://github.com/apache/spark/pull/35855#issuecomment-1079595660
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834509809
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to resolve as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ val analyzed = plan match {
Review comment:
Nice, that's more concise
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834952524
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2805,6 +2805,17 @@ object SQLConf {
.booleanConf
.createWithDefault(sys.env.get("SPARK_ANSI_SQL_MODE").contains("true"))
+ val ENABLE_DEFAULT_COLUMNS =
Review comment:
Sure, this is done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dtenedor commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
dtenedor commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r835455469
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.analysis.Analyzer
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #35855: [SPARK-38336][SQL] Support DEFAULT column values in CREATE/REPLACE TABLE statements
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #35855:
URL: https://github.com/apache/spark/pull/35855#discussion_r834422216
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDefaultColumns.scala
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.catalog._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser, ParseException}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types._
+
+/**
+ * This object contains fields to help process DEFAULT columns.
+ */
+object ResolveDefaultColumns {
+ // This column metadata indicates the default value associated with a particular table column that
+ // is in effect at any given time. Its value begins at the time of the initial CREATE/REPLACE
+ // TABLE statement with DEFAULT column definition(s), if any. It then changes whenever an ALTER
+ // TABLE statement SETs the DEFAULT. The intent is for this "current default" to be used by
+ // UPDATE, INSERT and MERGE, which evaluate each default expression for each row.
+ val CURRENT_DEFAULT_COLUMN_METADATA_KEY = "CURRENT_DEFAULT"
+ // This column metadata represents the default value for all existing rows in a table after a
+ // column has been added. This value is determined at time of CREATE TABLE, REPLACE TABLE, or
+ // ALTER TABLE ADD COLUMN, and never changes thereafter. The intent is for this "exist default"
+ // to be used by any scan when the columns in the source row are missing data. For example,
+ // consider the following sequence:
+ // CREATE TABLE t (c1 INT)
+ // INSERT INTO t VALUES (42)
+ // ALTER TABLE t ADD COLUMNS (c2 INT DEFAULT 43)
+ // SELECT c1, c2 FROM t
+ // In this case, the final query is expected to return 42, 43. The ALTER TABLE ADD COLUMNS command
+ // executed after there was already data in the table, so in order to enforce this invariant,
+ // we need either (1) an expensive backfill of value 43 at column c2 into all previous rows, or
+ // (2) indicate to each data source that selected columns missing data are to generate the
+ // corresponding DEFAULT value instead. We choose option (2) for efficiency, and represent this
+ // value as the text representation of a folded constant in the "EXISTS_DEFAULT" column metadata.
+ val EXISTS_DEFAULT_COLUMN_METADATA_KEY = "EXISTS_DEFAULT"
+ // Name of attributes representing explicit references to the value stored in the above
+ // CURRENT_DEFAULT_COLUMN_METADATA.
+ val CURRENT_DEFAULT_COLUMN_NAME = "DEFAULT"
+
+ /**
+ * Finds "current default" expressions in CREATE/REPLACE TABLE columns and constant-folds them.
+ *
+ * The results are stored in the "exists default" metadata of the same columns. For example, in
+ * the event of this statement:
+ *
+ * CREATE TABLE T(a INT, b INT DEFAULT 5 + 5)
+ *
+ * This method constant-folds the "current default" value, stored in the CURRENT_DEFAULT metadata
+ * of the "b" column, to "10", storing the result in the "exists default" value within the
+ * EXISTS_DEFAULT metadata of that same column. Meanwhile the "current default" metadata of this
+ * "b" column retains its original value of "5 + 5".
+ *
+ * @param tableSchema represents the names and types of the columns of the statement to process.
+ * @param statementType name of the statement being processed, such as INSERT; useful for errors.
+ * @return a copy of `tableSchema` with field metadata updated with the constant-folded values.
+ */
+ def constantFoldCurrentDefaultsToExistDefaults(
+ tableSchema: StructType, statementType: String): StructType = {
+ if (!SQLConf.get.enableDefaultColumns) {
+ return tableSchema
+ }
+ val newFields: Seq[StructField] = tableSchema.fields.map { field =>
+ if (field.metadata.contains(EXISTS_DEFAULT_COLUMN_METADATA_KEY)) {
+ val analyzed: Expression = analyze(field, statementType, foldConstants = true)
+ val newMetadata: Metadata = new MetadataBuilder().withMetadata(field.metadata)
+ .putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, analyzed.sql).build()
+ field.copy(metadata = newMetadata)
+ } else {
+ field
+ }
+ }
+ StructType(newFields)
+ }
+
+ /**
+ * Parses and analyzes the DEFAULT column text in `field`, returning an error upon failure.
+ *
+ * @param field represents the DEFAULT column value whose "default" metadata to parse and analyze.
+ * @param statementType which type of statement we are running, such as INSERT; useful for errors.
+ * @param foldConstants if true, perform constant-folding on the analyzed value before returning.
+ * @return Result of the analysis and constant-folding operation.
+ */
+ private def analyze(
+ field: StructField, statementType: String, foldConstants: Boolean = false): Expression = {
+ // Parse the expression.
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ val parsed: Expression = try {
+ lazy val parser = new CatalystSqlParser()
+ parser.parseExpression(colText)
+ } catch {
+ case ex: ParseException =>
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to parse as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ // Analyze the parse result.
+ val plan = try {
+ lazy val analyzer =
+ new Analyzer(new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin))
+ val analyzed = analyzer.execute(Project(Seq(Alias(parsed, field.name)()), OneRowRelation()))
+ analyzer.checkAnalysis(analyzed)
+ if (foldConstants) ConstantFolding(analyzed) else analyzed
+ } catch {
+ case ex @ (_: ParseException | _: AnalysisException) =>
+ val colText: String = if (field.metadata.contains(CURRENT_DEFAULT_COLUMN_METADATA_KEY)) {
+ field.metadata.getString(CURRENT_DEFAULT_COLUMN_METADATA_KEY)
+ } else {
+ "NULL"
+ }
+ throw new AnalysisException(
+ s"Failed to execute $statementType command because the destination table column " +
+ s"${field.name} has a DEFAULT value of $colText which fails to resolve as a valid " +
+ s"expression: ${ex.getMessage}")
+ }
+ val analyzed = plan match {
Review comment:
How about using
```
plan collectFirst
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org