You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "robert3005 (via GitHub)" <gi...@apache.org> on 2025/07/16 10:36:10 UTC
[I] BatchDataReader never closes CloseableIterable it creates [iceberg]
robert3005 opened a new issue, #13567:
URL: https://github.com/apache/iceberg/issues/13567
### Apache Iceberg version
1.9.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
BatchDataReader#open returns a CloseableIterator, however, this iterator is obtained from BaseBatchReader#newBatchIterable https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/BatchDataReader.java#L108-L116. The created CloseableIterable is immediately discarded and never closed. Whoever extends BaseBatchReader has to make sure to override open or remember to never register any resources on iterable returned from newBatchIterable.
You can see that OrcIterable suffers from this and will never close the underlying OrcFile from https://github.com/apache/iceberg/blob/main/orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java#L83-L84. Parquet doesn't suffer from this issue since the ParquetReader is added both to the CloseableIterable and CloseableIterator that's returned.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BatchDataReader never closes CloseableIterable it creates [iceberg]
Posted by "robert3005 (via GitHub)" <gi...@apache.org>.
robert3005 closed issue #13567: BatchDataReader never closes CloseableIterable it creates
URL: https://github.com/apache/iceberg/issues/13567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BatchDataReader never closes CloseableIterable it creates [iceberg]
Posted by "robert3005 (via GitHub)" <gi...@apache.org>.
robert3005 commented on issue #13567:
URL: https://github.com/apache/iceberg/issues/13567#issuecomment-3265949510
https://github.com/apache/iceberg/pull/12774 should hopefully address this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] BatchDataReader never closes CloseableIterable it creates [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #13567:
URL: https://github.com/apache/iceberg/issues/13567#issuecomment-3082306093
I think it's acceptable when `CloseableIterable` has no resources to close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org