You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "thomas-pfeiffer (via GitHub)" <gi...@apache.org> on 2026/03/19 21:02:41 UTC
[I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
thomas-pfeiffer opened a new issue, #3168:
URL: https://github.com/apache/iceberg-python/issues/3168
### Apache Iceberg version
0.11.0 (latest release)
### Please describe the bug 🐞
We have a table with a column containing large JSON strings (basically think of large `pydantic` validation results). The resulting datafile in Iceberg itself is a roughly 6MB (GZip compressed) Parquet file, but when querying it, the memory consumption goes to ~4GB. (Just for a few seconds, but long enough to cause out-of-memory issues on some systems.)
The issue is that by default `pyarrow` loads the strings per row into memory, which blows up the memory. If we download the datafile and open it directly via `pyarrow` this behaviour can be reproduced.
There are 2 workarounds in `pyarrow`: 1. Don't load the problematic column (given that is possible in your use case) and 2. switch to dictionary-encoding for set column (example snippet below).
```py
from pyarrow.parquet import read_table
table = read_table("datafile.parquet", read_dictionary=["problematic_column"])
```
Issue in `pyiceberg`:
Regardless if you use `table.scan(...).to_arrow()` or `table.scan(...).to_arrow_batch_reader()`, `pyiceberg` has afaik currently no option to specify the dictionary encoding for certain tables, hence `pyarrow` uses the default encoding and the memory usage explodes.
The `to_arrow_batch_reader` does not help here either, because -as per my understanding- in the batch reader of `pyiceberg` each batch represents an individual datafile. Hence, if there is one problematic 6MB datafile, it makes no difference if you use the batch reader or not. I also have the impression that when you iterate over the reader, `pyarrow` has already loaded the parquet file in a separate thread and this is where the memory explosion actually happens.
So the current only workaround in `pyiceberg` is option 1: Don't load the problematic column by specifying the `selected_fields`:
```py
from pyiceberg.catalog import load_catalog
catalog = load_catalog("default")
table = catalog.load_table("your_table")
reader = table.scan(selected_fields=("all", "other", "columns")).to_arrow_batch_reader()
for batch in reader:
...
```
Expected behaviour:
There should be an option somewhere, e.g. in the data_scan to specify for which columns dictionary encoding should be used. This option should be forwarded to `pyarrow` internally somehow, so that `pyarrow` uses less memory.
Remark:
I would not change the default behaviour. It would be just good to have the option to configure the encoding in pyarrow when needed.
This issue is a follow up for https://github.com/apache/iceberg-python/issues/1205
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [x] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #3168: When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow
URL: https://github.com/apache/iceberg-python/issues/3168
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3168:
URL: https://github.com/apache/iceberg-python/issues/3168#issuecomment-4100364150
> The issue is that by default pyarrow loads the strings per row into memory, which blows up the memory. If we download the datafile and open it directly via pyarrow this behaviour can be reproduced.
👍 #2676 solves the memory issue for reading multiple batches. If reading a single batch blows up the memory, the only fix here is to use dictionary-encoding
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3168:
URL: https://github.com/apache/iceberg-python/issues/3168#issuecomment-4100157344
Thanks for reporting this!
> The to_arrow_batch_reader does not help here either, because -as per my understanding- in the batch reader of pyiceberg each batch represents an individual datafile. Hence, if there is one problematic 6MB datafile, it makes no difference if you use the batch reader or not. I also have the impression that when you iterate over the reader, pyarrow has already loaded the parquet file in a separate thread and this is where the memory explosion actually happens.
This is a bug. I've found out about this recently (see https://github.com/apache/iceberg-python/discussions/3122). `to_arrow_batch_reader` should read from a single parquet file 1 batch at a time; thus reducing the memory consumption.
https://github.com/apache/iceberg-python/pull/2676 is the proper fix to correct this behavior. Would love to see if this helps with your issue.
> There should be an option somewhere, e.g. in the data_scan to specify for which columns dictionary encoding should be used. This option should be forwarded to pyarrow internally somehow, so that pyarrow uses less memory.
I think thats a reasonable feature request. Ive opened https://github.com/apache/iceberg-python/issues/3170 to track this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3168:
URL: https://github.com/apache/iceberg-python/issues/3168#issuecomment-5706563333
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "thomas-pfeiffer (via GitHub)" <gi...@apache.org>.
thomas-pfeiffer commented on issue #3168:
URL: https://github.com/apache/iceberg-python/issues/3168#issuecomment-4101517500
Adding here the documentation for the `read_dictionary` option of `pyarrow`: https://arrow.apache.org/docs/python/parquet.html#reading-types-as-dictionaryarray
I also think that the place where this additional parameter option would be needed is at:
https://github.com/apache/iceberg-python/blob/536a7d07100251fbe3f1074b9f5e7cf82a548d02/pyiceberg/io/pyarrow.py#L1100
I guess we would need the possibility to add something like the statement below - "just" passed through all of `pyiceberg`:
```py
ds.ParquetFileFormat(read_options=ds.ParquetReadOptions(dictionary_columns=["column1", "column2"]))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] When pyiceberg loads Iceberg tables containing large JSON data into memory, memory usage explodes in pyarrow [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #3168:
URL: https://github.com/apache/iceberg-python/issues/3168#issuecomment-5922339107
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org