You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Kesanov (via GitHub)" <gi...@apache.org> on 2023/02/21 11:59:55 UTC

[GitHub] [arrow] Kesanov opened a new issue, #34274: [Python] High RAM usage when reading parquet.

Kesanov opened a new issue, #34274:
URL: https://github.com/apache/arrow/issues/34274

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   My parquet file is 40GB in size. When loaded in memory, it is only 10GB in size.
   
   However, when reading from disk with `pyarrow.parquet.read_table`, the memory usage temporarily grows to 150GB.
   When reading from disk with `pandas.read_parquet(.., engine='pyarrow')`, the memory usage temporarily grows to 300GB.
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] westonpace commented on issue #34274: [Python] High RAM usage when reading parquet.

Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #34274:
URL: https://github.com/apache/arrow/issues/34274#issuecomment-1446912040

   If you have a `pyarrow.Table` (e.g. from a call to `pyarrow.parquet.read_table`) you can get a pretty precise data size with [`pyarrow.Table.nbytes`](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.nbytes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] Kesanov commented on issue #34274: [Python] High RAM usage when reading parquet.

Posted by "Kesanov (via GitHub)" <gi...@apache.org>.
Kesanov commented on issue #34274:
URL: https://github.com/apache/arrow/issues/34274#issuecomment-1445300583

   Yeah, I think you are right. I used `dataframe.info()`, which said the memory usage is `10G+`, which I interpreted as 10G exactly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] Kesanov closed issue #34274: [Python] High RAM usage when reading parquet.

Posted by "Kesanov (via GitHub)" <gi...@apache.org>.
Kesanov closed issue #34274: [Python] High RAM usage when reading parquet.
URL: https://github.com/apache/arrow/issues/34274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on issue #34274: [Python] High RAM usage when reading parquet.

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34274:
URL: https://github.com/apache/arrow/issues/34274#issuecomment-1439053918

   @Kesanov it sounds unlikely that the actual size in memory is only 10GB, since typically parquet files are very well compressed and tend to be much smaller on disk (but this depends on the exact way the file was written). So the 150GB doesn't sound unexpected to me for a file of 40GB.
   
   How did you measure the size in memory? Can you show some example code?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] jorisvandenbossche commented on issue #34274: [Python] High RAM usage when reading parquet.

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #34274:
URL: https://github.com/apache/arrow/issues/34274#issuecomment-1447933460

   > I used `dataframe.info()`, which said the memory usage is `10G+`
   
   If you have object dtype columns (for example string columns), this can be a large under-estimation. You can pass `dataframe.info(memory_usage="deep")` to get the full memory usage for the pandas.DataFrame.
   
   This can also be more than the memory usage you see for the pyarrow.Table (using `table.nbytes` as Weston mentioned above), since for some data types (such as strings), pandas is less efficient compared to pyarrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org