[GitHub] [arrow] jorisvandenbossche commented on issue #10965: Closing files after pyarrow.parquet read

[GitBox <gi...@apache.org>]

jorisvandenbossche commented on issue #10965:
URL: https://github.com/apache/arrow/issues/10965#issuecomment-903584702


   The ParquetDataset uses ParquetDatasetPiece to read the file(s), and that in turn uses `pq.ParquetFile` to open and read the actual Parquet file. And indeed, it seems we open but never close the file there. ParquetDatasetPiece also has an `open_file_func` which uses the `open` method from the filesystem, but also here this is used as a plain function (and not in a with context that would also close the file again) and we never explicitly close the file. 
   
   For `pq.read_table`, which version of pyarrow are you using? (or can you try with `use_legacy_dataset=False` to ensure to check with the newer implementation that doesn't use `pq.ParquetFile`)
   
   I see the issue when directly using `pq.ParquetFile` (checking that the file is still opened with `lsof`), but I don't directly see it with `pq.read_table` or `pq.ParquetDataset` using a simple example.
   
   To further discuss this, would you like to open a JIRA? (https://issues.apache.org/jira/projects/ARROW/issues/, which we use for tracking bug reports / feature requests)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org