You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Pear0 (via GitHub)" <gi...@apache.org> on 2025/10/20 02:49:09 UTC
[PR] [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Pear0 opened a new pull request, #47860:
URL: https://github.com/apache/arrow/pull/47860
### Rationale for this change
While arrow is designed for tall tables, sometimes I end up with a really wide table and the `to_pandas()` code seems to use N^2 memory where N is the number of extension array columns.
For example, the following snippet peaks at about 7GB of memory used on my machine:
```python
import pyarrow as pa
import pandas as pd
t = pa.table({f'col_{i}': pa.array([], type=pa.int64()) for i in range(10000)})
# with just this line, the process uses 7GB of memory at peak
t.to_pandas(types_mapper={pa.int64(): pd.ArrowDtype(pa.int64())}.get)
# with just this line, the process uses 118MB of memory at peak
t.to_pandas()
````
With this change, the extension array variation takes ~192MB of memory.
From what I can tell, this is because the `PandasOptions` struct is copied around frequently (for example it seems like there is an `ExtensionWriter` for each extension column and each `ExtensionWriter` has a copy of `PandasOptions` which has a set of all extension columns). I haven't fully traced the PandasOptions structure, but it seems to get copied and modified in some codepaths so I have decided to put the column sets into a `std::shared_ptr` rather than pass around a `shared_ptr<PandasOptions>`.
### What changes are included in this PR?
The `PandasOptions` column sets have been swapped from `std::unordered_set<std::string>` to `std::shared_ptr<const std::unordered_set<std::string>>` and usages have been updated.
### Are these changes tested?
Yes. Also tested memory usage by hand.
### Are there any user-facing changes?
- `PandasOptions` - `categorical_columns` and `extension_columns` have changed. Helper accessor functions have been added.
**This PR contains a "Critical Fix".**
I suppose it is technically a critical fix because you can trivially OOM your process using the repro above and a larger number of columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #47860:
URL: https://github.com/apache/arrow/pull/47860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "Pear0 (via GitHub)" <gi...@apache.org>.
Pear0 commented on code in PR #47860:
URL: https://github.com/apache/arrow/pull/47860#discussion_r2446780136
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.h:
##########
@@ -112,11 +128,11 @@ struct PandasOptions {
bool decode_dictionaries = false;
// Columns that should be casted to categorical
- std::unordered_set<std::string> categorical_columns;
+ std::shared_ptr<const std::unordered_set<std::string>> categorical_columns;
Review Comment:
I originally wanted to put the `PandasOptions` in a shared_ptr, but there are several code paths that want to take a `PandasOptions` and evolve it in some way (for example `MakeInnerOptions()`, `ConvertChunkedArrayToPandas()`) so it seems like you would still end up copying the sets pretty often in some circumstances (eg. I think once for each struct column?) unless you add indirection for the sets somehow.
If you'd still prefer not having the set indirection, I can take a stab at implementing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47860:
URL: https://github.com/apache/arrow/pull/47860#discussion_r2444051012
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.h:
##########
@@ -112,11 +128,11 @@ struct PandasOptions {
bool decode_dictionaries = false;
// Columns that should be casted to categorical
- std::unordered_set<std::string> categorical_columns;
+ std::shared_ptr<const std::unordered_set<std::string>> categorical_columns;
Review Comment:
Rather than adding `shared_ptr` indirections, the right solution IMHO is to avoid copying `PandasOptions`. `ConvertTableToPandas` should simply pass it by reference instead of by value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47860:
URL: https://github.com/apache/arrow/pull/47860#discussion_r2447138233
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.h:
##########
@@ -112,11 +128,11 @@ struct PandasOptions {
bool decode_dictionaries = false;
// Columns that should be casted to categorical
- std::unordered_set<std::string> categorical_columns;
+ std::shared_ptr<const std::unordered_set<std::string>> categorical_columns;
Review Comment:
Hmm, that's a good point, thank you. Then I agree we can keep the `std::shared_ptr`, and these are internal APIs anyway, so it doesn't matter if they do not look very elegant :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "Pear0 (via GitHub)" <gi...@apache.org>.
Pear0 commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3427619325
I've updated the PR with the comment suggestion and reworded the user-facing changes part of the description. I'm not very familiar with the pyarrow binding code. Feel free to reword it if you feel the PR description is still inaccurate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3420360823
:warning: GitHub issue #47861 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47860:
URL: https://github.com/apache/arrow/pull/47860#discussion_r2447156330
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.h:
##########
@@ -112,11 +128,11 @@ struct PandasOptions {
bool decode_dictionaries = false;
// Columns that should be casted to categorical
- std::unordered_set<std::string> categorical_columns;
+ std::shared_ptr<const std::unordered_set<std::string>> categorical_columns;
Review Comment:
Can we just add a comment to explain the rationale? For example:
```suggestion
//
// This is wrapped in a shared_ptr because this struct is copied internally for
// each column or nested field (see GH-47861).
std::shared_ptr<const std::unordered_set<std::string>> categorical_columns;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3457554036
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 57f626189326251c8144d4fad89c28220c68ca75.
There was 1 benchmark result indicating a performance regression:
- Commit Run on `amd64-m5-4xlarge-linux` at [2025-10-28 14:27:19Z](https://conbench.ursa.dev/compare/runs/81c31039482b4b60a61595865fbfc5a9...bb53fbf6cc0c4379bbc43725892d8824/)
- [`partitioned-dataset-filter` (R) with dataset=dataset-taxi-parquet, language=R, query=vignette](https://conbench.ursa.dev/compare/benchmarks/06900bd6d9fb74828000af5ed53732d7...06900d2ac95f7cd7800063c16e821877)
The [full Conbench report](https://github.com/apache/arrow/runs/53890935913) has more details. It also includes information about 11 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3420347981
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "Pear0 (via GitHub)" <gi...@apache.org>.
Pear0 commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3444980623
@pitrou btw I accepted your comment suggestion. Could you take another look at this PR when you get a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47861: [Python] reduce memory usage when using to_pandas() with many extension arrays columns [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #47860:
URL: https://github.com/apache/arrow/pull/47860#issuecomment-3426340692
@Pear0 thanks for the investigation and fix. Great catch!
> ### Are there any user-facing changes?
>
> * `PandasOptions` - `categorical_columns` and `extension_columns` have changed. Helper accessor functions have been added.
PandasOptions is only exposed in pyarrow C++ and not pyarrow itself (the cython wrapper is not publicly accessible I think), and while it technically is public in the `arrow::python`` C++ header, I assume that is not typically used externally, so the public facing part should be limited
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org