You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "UmashankarPalanichamy (via GitHub)" <gi...@apache.org> on 2025/11/23 06:53:16 UTC
[I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
UmashankarPalanichamy opened a new issue, #14667:
URL: https://github.com/apache/iceberg/issues/14667
### Apache Iceberg version
1.10.0 (latest release)
### Query engine
EMR
### Please describe the bug 🐞
On AWS EMR using Apache Iceberg Spark extensions, the rewrite_data_files stored procedure produces unexpected behavior when rewrite-all=true is used together with a where predicate.
When running:
CALL glue.system.rewrite_data_files(
table => 'glue.db.sample_table55',
strategy => 'sort',
sort_order => 'id ASC NULLS FIRST',
where => 'id IN (1,2)',
options => map(
'rewrite-all', 'true'
)
);
Iceberg generates a new data file containing the rewritten rows, but the previous data file continues to be referenced by the table snapshot. As a result, both the old and new files are scanned, producing duplicate rows.
Steps to Reproduce:
1. Create table:
CREATE TABLE glue.db.sample_table55 (id int, name string) USING iceberg;
2. Insert 4 rows:
Seq((1,"a"),(2,"b"),(3,"c"),(4,"d"))
.toDF("id","name")
.repartition(1)
.writeTo("glue.db.sample_table55")
.append();
3. Run rewrite_data_files:
CALL glue.system.rewrite_data_files(
table => 'glue.db.sample_table55',
strategy => 'sort',
sort_order => 'id ASC NULLS FIRST',
where => 'id IN (1,2)',
options => map('rewrite-all','true')
);
4. Query the table:
SELECT * FROM glue.db.sample_table55 ORDER BY id;
Actual result:
Rows appear duplicated because both old and new data files are referenced:
1 a
1 a
2 b
2 b
3 c
3 c
4 d
4 d
Expected behavior:
When rewrite-all=true is used, old data files should be fully replaced and not referenced in the new snapshot, unless partial-row preservation is required. Query results should not include duplicated rows from previous and rewritten data files.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "UmashankarPalanichamy (via GitHub)" <gi...@apache.org>.
UmashankarPalanichamy commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-3580463937
@hemanthboyina
Expected output
1 a
2 b
3 c
4 d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-3606059040
Hi @UmashankarPalanichamy I have tried to reproduce the scenario, but I am unable to in EMR 7.10.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "UmashankarPalanichamy (via GitHub)" <gi...@apache.org>.
UmashankarPalanichamy commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-3590252749
Hi @hemanthboyina
I recommend changing the rewrite-all + filter behavior so that the engine:
1. Scans only the files that match the where predicate,
2. Writes out new data files containing only the filtered/matching records, and
3. Explicitly removes the old data files either by
• issuing a delete operation for those files, or
• generating the corresponding position delete files for rows that were rewritten.
This ensures that the rewritten data fully replaces the old data and prevents the table from keeping both old and new files, which currently results in duplicated rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-4597711778
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-3580435313
Hi @UmashankarPalanichamy what should be the expected output based on the above query ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "UmashankarPalanichamy (via GitHub)" <gi...@apache.org>.
UmashankarPalanichamy commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-3606114768
Hi @hemanthboyina , thanks for checking this. Let's sync up on a quick call to discuss the details and make sure we're aligned on the reproduction steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14667:
URL: https://github.com/apache/iceberg/issues/14667#issuecomment-4713872591
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14667: rewrite_data_files with rewrite-all=true on EMR keeps referencing old data files, producing duplicate rows
URL: https://github.com/apache/iceberg/issues/14667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org