You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dataengineeratwork (via GitHub)" <gi...@apache.org> on 2026/07/02 03:30:18 UTC
[I] Merge Into Statement Lossing data on Non-partitioned table [iceberg]
dataengineeratwork opened a new issue, #17037:
URL: https://github.com/apache/iceberg/issues/17037
### Apache Iceberg version
1.4.0
### Query engine
Spark
### Please describe the bug 🐞
We've been using iceberg since long now on AWS, but came across the following issue recently in our Production environment.
With Upsert query using MERGE INTO stetement, we observed for one of the iceberg table (non-partitioned), while rewriting the data files, it has removed some of the records. We have checked with the time-travel and the records were present earlier, but in one the of rewrite history, the records got removed from the table.
The table here is in silver layer, where we're merging data from different data sources into single table, to identify the records, we've "datasource_id" column present along with some system columns like created_on_dt, updated_on_dt, w_inserted_dt, w_updated_dt, etl_run_id etc.
We've never observed any such issue with Partitioned tables though. Has anyone came across similar issue, I want to understand why iceberg is removing/deleting the unchanged records?
Following is the merge query we use :
```
upsert_sql = f"""
MERGE INTO {table} AS target
USING input_data AS source
ON concat(coalesce(target.integration_id, 'UNKNOWN'), coalesce(target.datasource_id, 'DFLT')) = concat(coalesce(source.integration_id, 'UNKNOWN'), coalesce(source.datasource_id, 'DFLT'))
WHEN MATCHED THEN
UPDATE SET {', '.join(f"target.{col} = source.{col}" for col in input_df.columns if col not in exclude_columns)}
WHEN NOT MATCHED THEN
INSERT ({', '.join(input_df.columns)})
VALUES ({', '.join(f"source.{col}" for col in input_df.columns)})
"""
# Execute the upsert SQL statement
spark.sql(upsert_sql)
cleaned_df = cleaned_df.dropDuplicates(["integration_id"])
upsert_data_iceberg(spark, cleaned_df, sl_athena_output_db_name, iceberg_table_name, s3_silver_output_location, logger)
```
Before snapshot:
snapshot-id = 5244992060615715741
total-records = 99,375,307
total-data-files = 936
Then the next snapshot:
snapshot-id = 7953544942377759370
added-records = 49,077,409
deleted-records = 55,719,302
added-data-files = 4
deleted-data-files = 38
total-records = 92,733,414
Net loss:
99,375,307
-92,733,414
------------
6,641,893 rows
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge Into Statement Lossing data on Non-partitioned table [iceberg]
Posted by "dataengineeratwork (via GitHub)" <gi...@apache.org>.
dataengineeratwork commented on issue #17037:
URL: https://github.com/apache/iceberg/issues/17037#issuecomment-4981492987
> > Apache Iceberg version
> > 1.4.0
>
> This seems to be a very old version, from October 4, 2023 (almost 3 years ago) https://iceberg.apache.org/releases/#140-release
>
> Are you able to test with a newer library version?
No, because we're using iceberg in AWS Glue jobs (v4.0), iceberg tables get created with version-2.
And the apache iceberg version is 1.4.2 I guess, that's what aws uses internally when we add datalake-format=iceberg in glue job properties.
I want to understand why it happened at first place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge Into Statement Lossing data on Non-partitioned table [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #17037:
URL: https://github.com/apache/iceberg/issues/17037#issuecomment-4981524366
There have been a lot of fixes in both Spark and Iceberg since then so it would be hard to tell exactly what was the cause. It's best to check if the same thing repos on a recent version of the library (and with a more recent spark.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge Into Statement Lossing data on Non-partitioned table [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #17037:
URL: https://github.com/apache/iceberg/issues/17037#issuecomment-4862603373
> Apache Iceberg version
1.4.0
This seems to be a very old version, from October 4, 2023 (almost 3 years ago) https://iceberg.apache.org/releases/#140-release
Are you able to test with a newer library version?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org