You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2024/01/24 17:43:55 UTC
[I] Spark 3.4 MERGE INTO for CoW replacing NULL unmatched records with default values [iceberg]
amogh-jahagirdar opened a new issue, #9555:
URL: https://github.com/apache/iceberg/issues/9555
### Apache Iceberg version
1.4.3 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Reproduction:
Here's a simple unit test (can copy/paste this into `TestMerge`)
```
@Test
public void testMergeIntoTsIssue() {
createAndInitTable(
"id INT, ts TIMESTAMP",
"{ \"id\": 1, \"ts\": \"2000-01-01 00:00:00\" }\n" + "{ \"id\": 6, \"ts\": null }");
createOrReplaceView(
"source",
"id INT NOT NULL, dep STRING",
"{ \"id\": 1, \"ts\": \"2000-01-01 00:00:00\" }\n");
sql(
"MERGE INTO %s t USING source s "
+ "ON t.id == s.id "
+ "WHEN MATCHED THEN "
+ " UPDATE SET id=123, ts=current_timestamp()",
commitTarget());
sql("SELECT * FROM %s", commitTarget());
}
```
In short:
1.) create a table
2.) insert some records where at least one of the records has a NULL column value.
3.) MERGE into the table with an update on matched records and set the column with the null value
Expected:
Record 1: id=123, ts=current_timestamp()
Record 2: id=6, ts=null
However, in Spark 3.4 we get
Record 1:id=123, ts=current_timestamp()
Record 2: id=6, ts=01-01-1970 00:00:000 (basically Unix epoch. in practice it's timestamp with tz so it'll appear to your timezone)
I've done some debugging and what's happening is that the schema for the `SparkWrite` in 3.4 is treating all the fields as required, leading to the default behavior.
The reason why it's treating it as required is because the Spark expression `nullability` is that the attributes for the fields https://github.com/apache/iceberg/blob/main/spark/v3.4/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala#L182 aren't being passed to the merge output schema during planning. This nullability needs to be passed correctly so that the null values in the non-matched cases get written correctly.
I'm currently looking into this, but creating this issue for tracking and awareness.
Important Note:
Spark 3.3 and Spark 3.5 do not have this bug based on my testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark 3.4 MERGE INTO for CoW replacing NULL unmatched records with default values [iceberg]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue closed issue #9555: Spark 3.4 MERGE INTO for CoW replacing NULL unmatched records with default values
URL: https://github.com/apache/iceberg/issues/9555
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org