You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jordi-crespo (via GitHub)" <gi...@apache.org> on 2025/09/01 23:04:13 UTC
[I] How to handle ValidationException: Cannot determine history during rewrite_data_files with concurrent streaming writes? [iceberg]
jordi-crespo opened a new issue, #13972:
URL: https://github.com/apache/iceberg/issues/13972
### Query engine
# Description:
I'm experiencing issues with Iceberg's rewrite_data_files procedure when trying to compact heavily fragmented tables while concurrent streaming writes are happening. Here's my environment:
- Apache Iceberg 1.9.2
- Nessie 0.10
- Apache Spark 3.5.2
- Streaming writes every few minutes
- Table partitioned by days(time), entity, and bucket(8, _id_asset)
# The Problem:
Some partitions have extreme fragmentation (e.g., 1295 files for just 2.5MB of data). When I try to compact these using rewrite_data_files, I consistently get:
````
ValidationException: Cannot determine history between starting snapshot 1456665297449387166 and the last known ancestor 6339655410305960251
````
# What I've Tried:
1. Using partial-progress.enabled='false' with retry logic (6 attempts with exponential backoff)
2. Using partial-progress.enabled='true'
3. Compactating at different granularities (by individual assets, by date ranges)
4. Adjusting various compaction parameters
# Additional Context:
The streaming process writes data every few minutes, and I need to regularly compact partitions that become heavily fragmented (1000+ files for just a few MB of data). The errors occur specifically on the most fragmented partitions.
Any guidance would be greatly appreciated!
### Question
1. What is the recommended approach for handling compaction with concurrent streaming writes?
2. Is there a way to make rewrite_data_files more resilient to concurrent modifications?
3. Should I be using different isolation levels or snapshot management techniques?
4. Is pausing streaming writes the only reliable solution for severely fragmented partitions?
5. Are there any planned improvements for automatic compaction in scenarios with frequent writes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to handle ValidationException: Cannot determine history during rewrite_data_files with concurrent streaming writes? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13972:
URL: https://github.com/apache/iceberg/issues/13972#issuecomment-4061797889
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to handle ValidationException: Cannot determine history during rewrite_data_files with concurrent streaming writes? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13972:
URL: https://github.com/apache/iceberg/issues/13972#issuecomment-3978701831
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to handle ValidationException: Cannot determine history during rewrite_data_files with concurrent streaming writes? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #13972: How to handle ValidationException: Cannot determine history during rewrite_data_files with concurrent streaming writes?
URL: https://github.com/apache/iceberg/issues/13972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org