[GitHub] [incubator-iceberg] vrozov commented on issue #272: Refactor AvroIterable and prevent multiple iterators

[GitBox <gi...@apache.org>]

vrozov commented on issue #272: Refactor AvroIterable and prevent multiple iterators
URL: https://github.com/apache/incubator-iceberg/pull/272#issuecomment-509728822
 
 
   The change helps to avoid resource shortage: as it is `AvroIterable` that is `Closeable`, not the `Iterator`, resources acquired in the `iterator()` won't be closed until `AvroIterable` itself is closed and in the case where multiple iterators are obtained from the same `AvroIterable`, there will be multiple file handles/sockets/http connections acquired and pointing to the same file/bucket that can't be closed individually.
   
   Note that `org.apache.avro.file.DataFileReader` follows a similar pattern. It is not possible to get multiple iterators from `DataFileReader` as it always returns the same iterator instance (itself).
   
   @rdblue What will be a use case were support for multiple iterators is necessary and that can't be covered by creating multiple `AvroIterable` instead?
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org