You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2025/02/21 18:53:53 UTC
[PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
aokolnychyi opened a new pull request, #50044:
URL: https://github.com/apache/spark/pull/50044
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
This PR enables filling default values in DSv2 writes.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
These changes are needed for proper support of default values for DSv2 connectors.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Users will be able to omit columns with default values. There is no impact to existing jobs.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
This patch comes with tests.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1978351011
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/V2WriteAnalysisSuite.scala:
##########
@@ -423,8 +423,8 @@ abstract class V2WriteAnalysisSuiteBase extends AnalysisTest {
assertNotResolved(parsedPlan)
assertAnalysisErrorCondition(
parsedPlan,
- expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA",
- expectedMessageParameters = Map("tableName" -> "`table-name`", "colName" -> "`x`")
+ expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.EXTRA_COLUMNS",
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1966310415
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/V2WriteAnalysisSuite.scala:
##########
@@ -423,8 +423,8 @@ abstract class V2WriteAnalysisSuiteBase extends AnalysisTest {
assertNotResolved(parsedPlan)
assertAnalysisErrorCondition(
parsedPlan,
- expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA",
- expectedMessageParameters = Map("tableName" -> "`table-name`", "colName" -> "`x`")
+ expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.EXTRA_COLUMNS",
Review Comment:
This is because of `spark.sql.defaultColumn.useNullsForMissingDefaultValues` and is aligned with V1 writes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968038807
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3534,7 +3534,8 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
TableOutputResolver.suitableForByNameCheck(v2Write.isByName,
expected = v2Write.table.output, queryOutput = v2Write.query.output)
val projection = TableOutputResolver.resolveOutputColumns(
- v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf)
+ v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf,
+ supportColDefaultValue = true)
Review Comment:
You mean `supportColDefaultValue` is true or false doesn't matter for v2 here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1981849551
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -400,18 +401,23 @@ abstract class InMemoryBaseTable(
val sizeInBytes = numRows * rowSizeInBytes
val numOfCols = tableSchema.fields.length
- val dataTypes = tableSchema.fields.map(_.dataType)
- val colValueSets = new Array[util.HashSet[Object]](numOfCols)
+ val colValueSets = new Array[util.HashSet[Any]](numOfCols)
val numOfNulls = new Array[Long](numOfCols)
for (i <- 0 until numOfCols) {
- colValueSets(i) = new util.HashSet[Object]
+ colValueSets(i) = new util.HashSet[Any]
}
inputPartitions.foreach(inputPartition =>
inputPartition.rows.foreach(row =>
for (i <- 0 until numOfCols) {
- colValueSets(i).add(row.get(i, dataTypes(i)))
- if (row.isNullAt(i)) {
+ val field = tableSchema(i)
+ val colValue = if (i < row.numFields) {
+ row.get(i, field.dataType)
+ } else {
+ ResolveDefaultColumns.getExistenceDefaultValue(field)
Review Comment:
I updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #50044: [SPARK-51290][SQL] Enable filling default values in DSv2 writes
URL: https://github.com/apache/spark/pull/50044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1976793379
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/V2WriteAnalysisSuite.scala:
##########
@@ -423,8 +423,8 @@ abstract class V2WriteAnalysisSuiteBase extends AnalysisTest {
assertNotResolved(parsedPlan)
assertAnalysisErrorCondition(
parsedPlan,
- expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA",
- expectedMessageParameters = Map("tableName" -> "`table-name`", "colName" -> "`x`")
+ expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.EXTRA_COLUMNS",
Review Comment:
If `useNullsForMissingDefaultValues` is false, suppose the test works as before? Maybe we should keep original tests and add new ones?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1980732719
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -400,18 +401,23 @@ abstract class InMemoryBaseTable(
val sizeInBytes = numRows * rowSizeInBytes
val numOfCols = tableSchema.fields.length
- val dataTypes = tableSchema.fields.map(_.dataType)
- val colValueSets = new Array[util.HashSet[Object]](numOfCols)
+ val colValueSets = new Array[util.HashSet[Any]](numOfCols)
val numOfNulls = new Array[Long](numOfCols)
for (i <- 0 until numOfCols) {
- colValueSets(i) = new util.HashSet[Object]
+ colValueSets(i) = new util.HashSet[Any]
}
inputPartitions.foreach(inputPartition =>
inputPartition.rows.foreach(row =>
for (i <- 0 until numOfCols) {
- colValueSets(i).add(row.get(i, dataTypes(i)))
- if (row.isNullAt(i)) {
+ val field = tableSchema(i)
+ val colValue = if (i < row.numFields) {
+ row.get(i, field.dataType)
+ } else {
+ ResolveDefaultColumns.getExistenceDefaultValue(field)
Review Comment:
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #50044:
URL: https://github.com/apache/spark/pull/50044#issuecomment-2702534394
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968032448
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -80,7 +80,6 @@ object TableOutputResolver extends SQLConfHelper with Logging {
query: LogicalPlan,
byName: Boolean,
conf: SQLConf,
- // TODO: Only DS v1 writing will set it to true. We should enable in for DS v2 as well.
Review Comment:
nit: in/it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968132329
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala:
##########
@@ -328,7 +336,7 @@ trait AlterTableTests extends SharedSparkSession with QueryErrorsBase {
}
test("SPARK-39383 DEFAULT columns on V2 data sources with ALTER TABLE ADD/ALTER COLUMN") {
- withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format, ") {
+ withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format,$catalog") {
Review Comment:
I think that it is because https://github.com/apache/spark/pull/50044/files#r1968081855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1979878201
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -80,7 +80,6 @@ object TableOutputResolver extends SQLConfHelper with Logging {
query: LogicalPlan,
byName: Boolean,
conf: SQLConf,
- // TODO: Only DS v1 writing will set it to true. We should enable in for DS v2 as well.
supportColDefaultValue: Boolean = false): LogicalPlan = {
Review Comment:
If I am not mistaken, there is one place that still invokes it with false [here](https://github.com/apache/spark/blob/921ef95609cde581d4ae32ab8ce18ec58913941d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala#L258).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1978057407
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala:
##########
@@ -263,4 +264,38 @@ class DataSourceV2DataFrameSuite
spark.listenerManager.unregister(listener)
}
}
+
+ test("columns with default values") {
+ val tableName = "testcat.ns1.ns2.tbl"
+ withTable(tableName) {
+ sql(s"CREATE TABLE $tableName (id INT, dep STRING) USING foo")
Review Comment:
nit: let's also test creating a V2 table with column default values, and then inserting into the V2 table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968042471
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3534,7 +3534,8 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
TableOutputResolver.suitableForByNameCheck(v2Write.isByName,
expected = v2Write.table.output, queryOutput = v2Write.query.output)
val projection = TableOutputResolver.resolveOutputColumns(
- v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf)
+ v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf,
+ supportColDefaultValue = true)
Review Comment:
Oh, I see. You mean to check for the flag `SUPPORT_COLUMN_DEFAULT_VALUE` here for the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968032448
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -80,7 +80,6 @@ object TableOutputResolver extends SQLConfHelper with Logging {
query: LogicalPlan,
byName: Boolean,
conf: SQLConf,
- // TODO: Only DS v1 writing will set it to true. We should enable in for DS v2 as well.
Review Comment:
nit: in/it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #50044: [SPARK-51290][SQL] Enable filling default values in DSv2 writes
URL: https://github.com/apache/spark/pull/50044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1978853315
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -400,18 +401,23 @@ abstract class InMemoryBaseTable(
val sizeInBytes = numRows * rowSizeInBytes
val numOfCols = tableSchema.fields.length
- val dataTypes = tableSchema.fields.map(_.dataType)
- val colValueSets = new Array[util.HashSet[Object]](numOfCols)
+ val colValueSets = new Array[util.HashSet[Any]](numOfCols)
val numOfNulls = new Array[Long](numOfCols)
for (i <- 0 until numOfCols) {
- colValueSets(i) = new util.HashSet[Object]
+ colValueSets(i) = new util.HashSet[Any]
}
inputPartitions.foreach(inputPartition =>
inputPartition.rows.foreach(row =>
for (i <- 0 until numOfCols) {
- colValueSets(i).add(row.get(i, dataTypes(i)))
- if (row.isNullAt(i)) {
+ val field = tableSchema(i)
+ val colValue = if (i < row.numFields) {
+ row.get(i, field.dataType)
+ } else {
+ ResolveDefaultColumns.getExistenceDefaultValue(field)
Review Comment:
It's not a big deal for now but given that we may want to make the test-only in-memory catalog to be a builtin catalog implementation for end users, I think it's better to make it right here.
It's very weird that we have to put the catalog name into the config `DEFAULT_COLUMN_ALLOWED_PROVIDERS `, as the catalog name is dynamic and specified by the users. We should let `getExistenceDefaultValue` take a bool parameter to indicate if existing default is supported or not, so that it's always true here, and we can set it to false for v1 data source w.r.t. the `DEFAULT_COLUMN_ALLOWED_PROVIDERS` config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1976098471
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3534,7 +3534,8 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
TableOutputResolver.suitableForByNameCheck(v2Write.isByName,
expected = v2Write.table.output, queryOutput = v2Write.query.output)
val projection = TableOutputResolver.resolveOutputColumns(
- v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf)
+ v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf,
+ supportColDefaultValue = true)
Review Comment:
Correct, I don't see value in checking `SUPPORT_COLUMN_DEFAULT_VALUE` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1967618587
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3534,7 +3534,8 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
TableOutputResolver.suitableForByNameCheck(v2Write.isByName,
expected = v2Write.table.output, queryOutput = v2Write.query.output)
val projection = TableOutputResolver.resolveOutputColumns(
- v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf)
+ v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf,
+ supportColDefaultValue = true)
Review Comment:
Yea true, Spark fills the default values during table writing and it works for all catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1966035022
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -718,6 +724,11 @@ private class BufferedRowsReader(
schema: StructType,
row: InternalRow): Any = {
val index = schema.fieldIndex(field.name)
+
+ if (index >= row.numFields) {
Review Comment:
This is needed for support for adding columns with default values to the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #50044:
URL: https://github.com/apache/spark/pull/50044#issuecomment-2675807959
cc @cloud-fan @szehon-ho @amaliujia @gengliangwang @dongjoon-hyun @viirya @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1978348982
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala:
##########
@@ -263,4 +264,38 @@ class DataSourceV2DataFrameSuite
spark.listenerManager.unregister(listener)
}
}
+
+ test("columns with default values") {
+ val tableName = "testcat.ns1.ns2.tbl"
+ withTable(tableName) {
+ sql(s"CREATE TABLE $tableName (id INT, dep STRING) USING foo")
Review Comment:
Sure, added a test below to verify CREATE/REPLACE statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1967623966
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala:
##########
@@ -328,7 +336,7 @@ trait AlterTableTests extends SharedSparkSession with QueryErrorsBase {
}
test("SPARK-39383 DEFAULT columns on V2 data sources with ALTER TABLE ADD/ALTER COLUMN") {
- withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format, ") {
+ withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format,$catalog") {
Review Comment:
How does this conf affect the testing v2 in-memory catalog? I thought it's only for v1 file source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1979905938
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -400,18 +401,23 @@ abstract class InMemoryBaseTable(
val sizeInBytes = numRows * rowSizeInBytes
val numOfCols = tableSchema.fields.length
- val dataTypes = tableSchema.fields.map(_.dataType)
- val colValueSets = new Array[util.HashSet[Object]](numOfCols)
+ val colValueSets = new Array[util.HashSet[Any]](numOfCols)
val numOfNulls = new Array[Long](numOfCols)
for (i <- 0 until numOfCols) {
- colValueSets(i) = new util.HashSet[Object]
+ colValueSets(i) = new util.HashSet[Any]
}
inputPartitions.foreach(inputPartition =>
inputPartition.rows.foreach(row =>
for (i <- 0 until numOfCols) {
- colValueSets(i).add(row.get(i, dataTypes(i)))
- if (row.isNullAt(i)) {
+ val field = tableSchema(i)
+ val colValue = if (i < row.numFields) {
+ row.get(i, field.dataType)
+ } else {
+ ResolveDefaultColumns.getExistenceDefaultValue(field)
Review Comment:
I agree it is awkward to set the catalog name in the SQL conf to enable default values. It was merely an improvement over the existing approach where we set a provider with a single space like [here](https://github.com/apache/spark/blob/921ef95609cde581d4ae32ab8ce18ec58913941d/sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala#L367).
The primary problem is that `CatalogV2Util$applySchemaChanges` used by the in-memory catalog requires the provider name and always checks `DEFAULT_COLUMN_ALLOWED_PROVIDERS`. Right now, we were passing an empty string, so we had to do that workaround in the SQL conf. In other words, it fails during ALTER statements.
One alternative can be to pass `in-memory` from the catalog implementation and modify `CatalogV2Util` to allow that provider if `Utils.isTesting` is true. That way, we won't have to modify the SQL conf for default values to work.
What do you think, @cloud-fan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #50044:
URL: https://github.com/apache/spark/pull/50044#issuecomment-2702697048
Thanks for reviewing, @cloud-fan @gengliangwang @dongjoon-hyun @viirya!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #50044: [SPARK-51290][SQL] Enable filling default values in DSv2 writes
URL: https://github.com/apache/spark/pull/50044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1966030259
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -3534,7 +3534,8 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
TableOutputResolver.suitableForByNameCheck(v2Write.isByName,
expected = v2Write.table.output, queryOutput = v2Write.query.output)
val projection = TableOutputResolver.resolveOutputColumns(
- v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf)
+ v2Write.table.name, v2Write.table.output, v2Write.query, v2Write.isByName, conf,
+ supportColDefaultValue = true)
Review Comment:
I don't think there is value in validating if the catalog defines `SUPPORT_COLUMN_DEFAULT_VALUE` in capabilities during writes. If a connector includes default value metadata in its schema, it should be enough to fill default values. The flag exists for ALTER and CREATE/REPLACE statements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968081855
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala:
##########
@@ -122,7 +122,7 @@ class BasicInMemoryTableCatalog extends TableCatalog {
override def alterTable(ident: Identifier, changes: TableChange*): Table = {
val table = loadTable(ident).asInstanceOf[InMemoryTable]
val properties = CatalogV2Util.applyPropertiesChanges(table.properties, changes)
- val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, None, "ALTER TABLE")
+ val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, Some(name), "ALTER TABLE")
Review Comment:
Do we need to add memory table provider to `DEFAULT_COLUMN_ALLOWED_PROVIDERS`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968084126
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala:
##########
@@ -122,7 +122,7 @@ class BasicInMemoryTableCatalog extends TableCatalog {
override def alterTable(ident: Identifier, changes: TableChange*): Table = {
val table = loadTable(ident).asInstanceOf[InMemoryTable]
val properties = CatalogV2Util.applyPropertiesChanges(table.properties, changes)
- val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, None, "ALTER TABLE")
+ val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, Some(name), "ALTER TABLE")
Review Comment:
Oh, nvm, the name is given when initializing the catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1968152464
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -718,6 +724,11 @@ private class BufferedRowsReader(
schema: StructType,
row: InternalRow): Any = {
val index = schema.fieldIndex(field.name)
+
+ if (index >= row.numFields) {
Review Comment:
This is method `extractFieldValue`. Looks like it is only used by `get`. Why this is for adding columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1976098170
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/AlterTableTests.scala:
##########
@@ -328,7 +336,7 @@ trait AlterTableTests extends SharedSparkSession with QueryErrorsBase {
}
test("SPARK-39383 DEFAULT columns on V2 data sources with ALTER TABLE ADD/ALTER COLUMN") {
- withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format, ") {
+ withSQLConf(SQLConf.DEFAULT_COLUMN_ALLOWED_PROVIDERS.key -> s"$v2Format,$catalog") {
Review Comment:
@viirya is correct. We previously passed `"" ` as provider in the in-memory connector, which required workarounds like this. No longer needed as we pass the catalog name as provider. Simplifies testing.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryTableCatalog.scala:
##########
@@ -122,7 +122,7 @@ class BasicInMemoryTableCatalog extends TableCatalog {
override def alterTable(ident: Identifier, changes: TableChange*): Table = {
val table = loadTable(ident).asInstanceOf[InMemoryTable]
val properties = CatalogV2Util.applyPropertiesChanges(table.properties, changes)
- val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, None, "ALTER TABLE")
+ val schema = CatalogV2Util.applySchemaChanges(table.schema, changes, Some(name), "ALTER TABLE")
Review Comment:
Correct.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/InMemoryBaseTable.scala:
##########
@@ -718,6 +724,11 @@ private class BufferedRowsReader(
schema: StructType,
row: InternalRow): Any = {
val index = schema.fieldIndex(field.name)
+
+ if (index >= row.numFields) {
Review Comment:
This is needed to read data inserted prior to adding columns to the schema. If that happens, there would be extra columns in the schema and we have to default new columns using the existence default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1978834631
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:
##########
@@ -80,7 +80,6 @@ object TableOutputResolver extends SQLConfHelper with Logging {
query: LogicalPlan,
byName: Boolean,
conf: SQLConf,
- // TODO: Only DS v1 writing will set it to true. We should enable in for DS v2 as well.
supportColDefaultValue: Boolean = false): LogicalPlan = {
Review Comment:
do we still need this flag if it's always true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #50044:
URL: https://github.com/apache/spark/pull/50044#discussion_r1977930078
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/V2WriteAnalysisSuite.scala:
##########
@@ -423,8 +423,8 @@ abstract class V2WriteAnalysisSuiteBase extends AnalysisTest {
assertNotResolved(parsedPlan)
assertAnalysisErrorCondition(
parsedPlan,
- expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.CANNOT_FIND_DATA",
- expectedMessageParameters = Map("tableName" -> "`table-name`", "colName" -> "`x`")
+ expectedErrorCondition = "INCOMPATIBLE_DATA_FOR_TABLE.EXTRA_COLUMNS",
Review Comment:
Sure, I am OK either way. Let me update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51290][SQL] Enable filling default values in DSv2 writes [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #50044:
URL: https://github.com/apache/spark/pull/50044#issuecomment-2693071459
@viirya @cloud-fan, could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org