Re: [I] [C++] Potential memory leak in Parquet reading with Dataset [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on issue #37630:
URL: https://github.com/apache/arrow/issues/37630#issuecomment-2607468363

   I've submitted a [draft PR](https://github.com/apache/arrow/pull/45330) that adds a `cache_metadata` option when scanning.
   
   Here are some memory consumption measurements on a synthetic dataset similar to the one in #45287:
   * with `cache_metadata=True` (the default):
   ```console
   $ /usr/bin/time -f "\n%E real\n%U user\n%S sys\n%M kB peak RSS" python  -c 'import pyarrow.dataset as pd; pd.dataset("/home/antoine/arrow/data/bamboo-streaming-parquet-test-data/1000col-dataset/").to_table()'
   
   0:11.01 real
   107.66 user
   4.37 sys
   2069284 kB peak RSS
   ```
   * with `cache_metadata=False`:
   ```console
   $ /usr/bin/time -f "\n%E real\n%U user\n%S sys\n%M kB peak RSS" python  -c 'import pyarrow.dataset as pd; pd.dataset("/home/antoine/arrow/data/bamboo-streaming-parquet-test-data/1000col-dataset/").to_table(cache_metadata=False)'
   
   0:10.97 real
   109.11 user
   3.87 sys
   1381696 kB peak RSS
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org