You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "stephen-a-stc (via GitHub)" <gi...@apache.org> on 2025/01/17 15:30:03 UTC
[I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
stephen-a-stc opened a new issue, #45296:
URL: https://github.com/apache/arrow/issues/45296
### Describe the bug, including details regarding any error messages, version, and platform.
## Summary
Using the pyarrow table method `to_pandas()` results in an exception if `pandas.set_option('future.infer_string', True)` has been set. This seems related to handling of string data.
## Environment
Environments: Windows 11, and Linux (docker image "python")
Python versions tested: 3.10, 3.11, 3.12
Python packages: `pyarrow==19.0.0`, `pandas==2.2.3`
## Example
```python
import pandas
import pyarrow
print("Create pyarrow table")
pat=pyarrow.Table.from_pydict({"foo":["bar","baz"]})
print(pat)
print("convert to pandas")
df1 = pat.to_pandas()
print(df1)
print("Set 'future.infer_string' to True")
pandas.set_option('future.infer_string', True)
print("exception during convert to pandas")
df2 = pat.to_pandas()
print(df2)
```
## Example's output
```plaintext
Create pyarrow table
pyarrow.Table
foo: string
----
foo: [["bar","baz"]]
convert to pandas
foo
0 bar
1 baz
Set 'future.infer_string' to True
convert to pandas again
Traceback (most recent call last):
File "C:\temp\foo.py", line 16, in <module>
df2 = pat.to_pandas()
File "pyarrow\\array.pxi", line 889, in pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow\\table.pxi", line 5132, in pyarrow.lib.Table._to_pandas
File "C:\penv\brd_test\lib\site-packages\pyarrow\pandas_compat.py", line 800, in table_to_dataframe
ext_columns_dtypes = _get_extension_dtypes(
File "C:\penv\brd_test\lib\site-packages\pyarrow\pandas_compat.py", line 925, in _get_extension_dtypes
ext_columns[field.name] = _pandas_api.pd.StringDtype(na_value=np.nan)
TypeError: StringDtype.__init__() got an unexpected keyword argument 'na_value'
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598642144
This is related to:
- https://github.com/apache/arrow/issues/43683
It seems the `na_value` was [merged on pandas main](https://github.com/pandas-dev/pandas/commit/f25a09eacee843f643604a7406a5bb6bcc4361e5) but is not available on pandas 2.2 (where `future.infer_string` was added)
@jorisvandenbossche @WillAyd do we have to check this is specifically for Pandas 3 or should we specialize the call based on pandas 2 vs 3?
https://github.com/apache/arrow/blob/5181c24f6d780591426e0af0d1b72c68f190484c/python/pyarrow/pandas_compat.py#L917-L925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2622275706
Opened a PR with my suggestion above so it can be included in the 19.0.1 release: https://github.com/apache/arrow/pull/45383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #45296: [Python] to_pandas() fails when pandas option 'future.infer_string' is True
URL: https://github.com/apache/arrow/issues/45296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598670892
In the meantime @stephen-a-stc you can either downgrade Arrow to anything less than version 19, or use a development version of pandas with Arrow 19+, and I think you will get the result you are after
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2622430866
Thanks @jorisvandenbossche!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598677632
Ah, yes, the current code is assuming that the user only uses the `'future.infer_string'` option when having pandas 2.3+ .. I should have included that in the check at:
https://github.com/apache/arrow/blob/c9f417f1a58663351692433559e85ff2a7ec2e19/python/pyarrow/pandas-shim.pxi#L182-L187
(checking for that option `and pd.__version__ >= 2.3.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2624348506
Issue resolved by pull request 45383
https://github.com/apache/arrow/pull/45383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598664937
Yea this is not expected to work with that option in pandas 2.2. The 2.3 release should resolve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598688797
> Yea this is not expected to work with that option in pandas 2.2. The 2.3 release should resolve it
Ah, ok, I wasn't sure this was going to be added to pandas 2.3. @jorisvandenbossche solution makes sense to me for future cases of people using pandas 2.2 and setting `infer_string` but I don't think we are currently planning a patch release for Arrow at the moment so as @WillAyd suggests using pyarrow < 19 until pandas 2.3 is released or dev pandas is the only solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] to_pandas() fails when pandas option 'future.infer_string' is True [arrow]
Posted by "stephen-a-stc (via GitHub)" <gi...@apache.org>.
stephen-a-stc commented on issue #45296:
URL: https://github.com/apache/arrow/issues/45296#issuecomment-2598683765
Okay, thanks for the information.
I can confirm that this wasn't an issue with earlier versions of pyarrow, for instance it works fine with `pyarrow==18.1.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org