[GitHub] [incubator-iceberg] rdblue commented on issue #104: ManifestReader is not properly closed in BaseTableScan

[GitBox <gi...@apache.org>]

rdblue commented on issue #104: ManifestReader is not properly closed in BaseTableScan
URL: https://github.com/apache/incubator-iceberg/issues/104#issuecomment-479639244
 
 
   @aokolnychyi, @prodeezy, I found the problem. Thanks for the tips that TestParquetWrite can reproduce. By adding a gc to the end of that test, I was able to reliably reproduce the problem.
   
   The problem was caused by `CloseableIterable.combine`. That method copied the list of closeables immediately, instead of referencing the original queue where the closeables were added as the scan progressed. I've replaced that with a `CloseableIterable.concat` operation that closes the iterables more quickly, as they are exhausted. I also updated `ParallelIterable` to do the same, so now manifests are closed after tasks finish instead of at the end of scan planning. This should be better for resource use.
   
   I also confirmed that I no longer see the unclosed stream warning for either code path (parallel or single-threaded).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org